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EXECUTIVE SUMMARY 


Project Exc-EL (Excellence for English Learners) is a school-wide intervention focused on 
providing teacher professional development and training to better support English learners (ELs). 
The ultimate goal of the intervention is to support the academic achievement and post-secondary 
success of EL students. To support EL students, Project Exc-EL focuses on training teachers to 
use data to identify EL student needs, creating a school climate focused on college and career 
readiness, and providing school coaches to support teachers as they serve EL students and their 
families. The core philosophy is one of enriched activities and wrap around supports focused on 
success, college/career readiness, and high school completion. Project participants used 
individualized student learning plans and tiered interventions to track and adjust student 
activities. 


Project Exc-EL developed and implemented an enhanced, comprehensive design that addressed 
the unique and urgent needs of low-incidence EL population school districts — districts that are 
struggling to provide a comprehensive, rigorous education for the newest members of their 
communities. The project employed a data-driven, tiered approach to instruction that builds on 
community partnerships to create personalized, expanded learning opportunities for students. 


External evaluators conducted an impact and an implementation study to ascertain the outcomes 
of the project and gauge the fidelity of implementation of the project. The evaluation questions 
address whether Project Exc-EL increases EL student achievement in math and English 
Language Arts (ELA), and how the developer implemented the program across four schools. 
Figure ES-1 shows the change over time in math and ELA outcomes for EL students one year 
prior to the intervention (baseline) and four years post-intervention. There were no statistically 
significant differences in math and ELA outcomes of EL students between the treatment and 
comparison schools. 


Figure ES-1: Impact Results for Math and English Language Arts for EL Students 
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Note: Data from publicly available school records from the New York Department of Education. There were 4 
treatment schools and 16 statistically matched comparison schools. 
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The project was implemented by the developer with fidelity all three years of project 
implementation, see Figure ES-2. Specifically, the developers implemented the key components 
related to creating a school culture of college and career readiness, training teachers in data use, 
and providing school coaches to support teachers. 


In all, Project Exc-EL was a five-year Investing in Innovation (i3) grant funded by the US 
Department of Education to provide professional development, training, and coaching to school 
staff to better support EL students. While the developers implemented the project with fidelity, 
there were no significant differences in math or ELA outcomes of EL students. Given that the 
focus is on training teachers on data use, building school culture, and providing school coaches 
for teachers, future research should include intermediate outcomes, or moderators, such as 
improved teacher instruction and pedagogy, to better understand the mechanisms of student 
achievement for English learners. 
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Figure ES-2: Implementation Results Project Exc-EL 
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1.1 PROJECT Exc-EL KEY COMPONENTS 


The University of California at Los Angeles’ Center X applied for and received funding for 
Project Exc-EL (Excellence for English Learners) via an Investing in Innovation (13) 
Development Grant in 2013. Project Exc-EL is a school-wide initiative that features 3 key 
components: 

e School climate and structures to support college and career readiness; 

e Teacher and staff training and technical assistance; and 

e Data-driven systematic coaching. 


These complementary components are designed to improve the college readiness rates and 
overall student outcomes of English learners (ELs). The college readiness rates of ELs are low 
when compared to the general population. ELs are often academically underprepared, resulting 
in significant achievement gaps. To better support EL students and their families as they prepare 
to graduate from high school and enter college, Project Exc-EL developers created a school-wide 
initiative to help schools better meet the needs of their EL population. 


School climate and structures to support college and career readiness 

Each school participating in Project Exc-EL had school coaches who helped to provide 
leadership and guidance on creating a school-wide college-ready culture. Topics of the coaching 
sessions included co-teaching, personalized learning settings, scheduling for teacher 
development and planning time, parent engagement, and reducing achievement gaps with 
additional wraparound student and family supports from both the school and community 
partners. Project Exc-EL developers assembled a partnership consisting of community 
organizations positioned to provide additional supports, such as tutoring, financial aid and 
college application assistance, and parent/family supports and training. These partners were 
focused on providing wrap-around supports to EL students and their families to further ensure 
that these students are college-ready. 


Teacher and staff training and technical assistance 

Project Exc-EL provided teachers and other school staff with specific training—two day Summer 
Institutes (SI) held each August over the life of the project. These institutes focused on classroom 
instructional supports and interventions, including Response to Intervention [RtI] and Dynamic 
Language Learning progressions [DLLP]). The institutes also provided time for project 
participants to share strategies and methods that helped discern and meet the needs of ELs and 
cross-school and cross-district. In addition to the institutes, school teams, teacher leaders, and EL 
students were supported as they attended and presented at nationally-known conferences on 
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topics relevant to ELs and their families. Conference attendees returned from these conferences 
and further disseminated the latest strategies among their peers. 


Data-driven systematic coaching 

Project Exc-EL worked with teachers and school staff grouped into Professional Learning 
Communities (PLCs). These PLCs met regularly to review EL student data and provide more 
direct one-on-one supports to EL students, while collaboratively planning for co-teaching. 
Coaches met with the PLCs to provide feedback, guidance as needed, and to connect the school 
teams to broader project resources as needed. 


1.2 PROJECT Exc-EL LoGic MODEL 


The evaluation team worked with the developer and the evaluation technical assistance provider 
to refine and further develop the project logic model contained in the grant application. The 
resulting logic model codifies the project and is available in Figure 1 below. 

e Project Inputs are listed in the left column of the logic model and include resources, staff, and 
partners necessary to implement the project. 

e The center column features the Project Exc-EL Key Components or core features of Project 
Exc-EL. Key components are the ideas and concepts at the heart of Exc-EL that are intended 
to affect educational practice at the school level. Each key component was used to develop 
fidelity of implementation indicators and definitions of these indicators (see Appendix A). 

e The right column, Mediators, lists the conduits or pathways that practices are expected to 
follow to ultimately manifest as Student Outcomes. The evaluation team used the student 
outcomes to discern which data would prove relevant to estimating the impact of Project 
Exc-EL over the life of the project. 
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Figure 1: Project Exc-EL Logic Model 
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1.3 EVALUATION OVERVIEW 


Impact Study 

Project Exc-EL is a school-level intervention focused on teacher training and development, 
school cultural changes, and teacher data teams. These key components were hypothesized to 
directly impact English learners (ELs) as they prepare for college and indirectly impact all 
students in the schools. Four schools (2 middle schools and 2 high schools from 2 Westchester 
County, New York school districts) implemented the intervention (treatment group). 


The impact study is a quasi-experimental design (QED), wherein we statistically matched 
schools to be comparison group schools (Shadish, Cook, & Campbell, 2002) (see Appendix A 
for details). Therefore, we have four treatment schools, with a carefully matched comparison 
group of 16 schools (Becker, 2002; Dehejia & Wahba, 2002; Rosembaum, 1984). We compared 
the school outcomes of the 4 treatment schools to the 16 comparison schools on mathematics 
achievement for EL students in the schools and English language arts achievement for EL 
students in the schools. The student outcome questions (SOQ) that guided the impact study were 
as follows: 
e SOQ I What is the impact of Project Exc-EL on the school-level mathematics achievement 
of English learners? 
e SOQ 2 What is the impact of Project Exc-EL on the school-level reading achievement of 
English learners? 


To estimate the impacts, we first identified the comparison schools through propensity score 
matching techniques and conducted baseline equivalence testing to ensure that the treatment and 
comparison schools were similar on key outcomes one-year prior to the intervention (See 
Appendix B for details). Second, our impact analytic approach included calculating the effect 
size based on the difference between the pre-test at baseline and post-test in Year 4, and 
analyzing 11 years of data based on a comparison short interrupted time series (C-ITS) design 
(Bloom, 2003) (See Appendix C for details). 


Implementation Study 

Plus Alpha worked with the project developer to design an implementation study that allowed 
the flexibility needed for a development grant while ensuring that fidelity to the key project 
components is defined and assessed across the treatment group (Nelson et al, 2012). It is 
important to note that fidelity of implementation measures the developer’s actions in regards to 
the center area of the logic model, the Project Exc-EL Key Components. 


Measuring Project Exc-EL fidelity began with refining the project logic model provided in the 


original grant application. This logic model guided the implementation study design. The logic 
model aligns with the management plan created by the developer and approved by the US 
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Department of Education (ED) Program Officer. Each key component consists of indicators of 
implementation, as can be seen in Appendix A in the Fidelity Matrices for each key component. 
We provided the operational definition for each indicator as well. 


We created protocols (see Appendix D) wherein each protocol item aligns with an indicator of 
implementation and therefore a key component. Each protocol item was scored either yes or no 
(0 or 1). Scores roll up to the school level and to the full sample. We used the Fidelity Matrix to 
measure and assess fidelity for all components and indicators for each of the three years of 
implementation. 


Implementation questions (IQ) guided the assessment of fidelity as follows: 
e JQ 1 Have the key components of Project Exc-EL been implemented with fidelity? 
e JQ 2 How has implementation varied across the treatment schools in terms of the key project 
components: 
o School climate and structures to support college and career readiness, 
o Teacher and staff training and technical assistance, and 
o Data-driven systemic coaching. 


Measuring fidelity helps to better define and ascertain what implementing Project Exc-EL with 
high fidelity entails. We designed a series of aligning documents to map from the project logic 
model to the project management plan objectives, strategies, and actions. A fidelity matrix has 
been designed to measure fidelity based on tangible developer-dependent activities and roles and 
score fidelity at both the school and treatment group levels (see Appendix A: Evaluation 
Methodology). Instruments and protocols have been created to obtain data annually from 
relevant project participants (see Appendix C: Implementation Study Protocols). Fidelity scoring 
and content analysis were used to measure the fidelity of implementation. We used these 
documents to guide data gathering in response to the implementation questions. 
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2.1 IMPACT STUDY 


The impact study features the full sample of 4 treatment schools (2 middle and 2 high schools 
across 2 districts in New York) and 16 comparison schools (8 middle and 8 high schools across 
New York) for a total of 20 schools in the analytic sample. The impact study compares math and 
English language arts achievement for English learner (EL) students', using school report card 
data.” 


Mathematics achievement was measured by the state assessment. For middle grades (grades 6-8), 
the New York State Assessment in mathematics was used, and for high school, the Regents 
Integrated Algebra assessment was used. English language arts achievement was measured by 
the state assessment. For middle grades (grades 6-8), the New York State Assessment in 
language arts was used, and for high school, the Regents Comprehensive English assessment was 
used. All achievement scores were converted into a standardized score (z-score), where zero is 
the state mean. Scores above the mean (zero) denote test scores that were above the state 
average. Scores below the mean (zero) denote test scores that were less than the state average. 


For graphical purposes, the figures below (Figure 2 and Figure 3) showcase five years of school- 

level data. 

e Baseline. The 2012-2013 school year, 1 year prior to implementing Project Exc-EL. 

e Year 1. The 2013-2014 school year, the first year that the 4 treatment schools were 
implementing Project Exc-EL. 

e Year 2. The 2014-2015 school year, the second year that the 4 treatment schools were 
implementing Project Exc-EL. 

e Year 3. The 2015-2016 school year, the third year that the 4 treatment schools were 
implementing Project Exc-EL. 

e Year 4. The 2016-2017 school year, the fourth year that the 4 treatment schools were 
implementing Project Exc-EL. 


To ensure that the 16 comparison schools were similar, we conducted a series of analyses to 
ensure that we were comparing like-to-like, or ‘apples-to-apples’. Similar schools are those 
schools in New York that were comparable demographically (i.e., percent LEP), academically 
(i.e., math and ELA performance of LEP students), and behaviorally (i.e., overall school 


' The New York State Department of Education refers to English learners (ELs) as “Limited English Proficient” 
(LEP) students. 


? School report card data was obtained through the New York State Department of Education, 
https://data.nysed.gov/downloads.php. 
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attendance).? In conducting baseline equivalence, the analytic sample meets What Works 
Clearinghouse (WWC) Evidence Standards for baseline equivalence, where all key variables had 
a standardized mean difference of less than .25. 


Figure 2 shows the math achievement results for EL students. At baseline, the treatment and 
comparison schools had similar math scores for EL students, within the .25 threshold as specified 
by the WWC Standards.* From baseline, there were no differences in math achievement of EL 
students between the treatment and comparison schools (Effect size = 0.08).>° 


Figure 2: Math Achievement for English Learner Students 
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Figure 3 shows the English language arts achievement results for EL students. At baseline, the 
treatment and comparison schools had similar ELA scores for EL students, within the .25 


3 According to the What Works Clearinghouse (WWC) Evidence Standards (version 4.0), baseline equivalence is 
met if the effect size of key outcomes is less than .25 (i.e., comparing ‘apples-to-apples’). Baseline equivalence 
is not met if key outcomes are over .25 (i.e., comparing ‘apples-to-oranges’.) 


4 The What Works Clearinghouse Standards 4.0 requires representativeness of the sample for cluster designs at 
baseline. The math outcome at baseline meets the optimistic boundary and meets WWC representativeness 
standards. 


> The evaluators also conducted an interrupted time series with comparison analyses for math achievement and 
found not significant impacts. See appendix C. 


© The What Works Clearinghouse Standards 4.0 requires representativeness of the sample for cluster designs at 
follow-up. The math outcome at follow does not meet the optimistic boundary and does not meet WWC 
representativeness standards. 
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threshold as specified by the WWC Standards.’ From baseline, there were no significant 
differences in ELA achievement of EL students between the treatment and comparison schools 
(Effect size = -0.18).°° However, it is interesting to note that while not statistically significant, 
the treatment schools had lower ELA achievement among their EL students after four years of 
treatment than the comparison schools, with a steady decline in the past three years. 


Figure 3: English Language Arts Achievement for English Learner Students 
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2.2 IMPLEMENTATION STUDY 


Based on data collected in all three implementation years of Project Exc-EL, as outlined in 
Appendix A, the developer has implemented Project Exc-EL with fidelity in three out of the four 
years, see Figure 4 below. Fidelity indicators were based on developer-dependent roles and 
responsibilities, so a finding of implemented with fidelity indicates that the developer has 
implemented strategies and activities as outlined in the annual project management plan. 


7 While ELA outcome meets baseline equivalence standards, the WWWC 4.0 now requires representativeness of the 
sample for cluster designs. The ELA outcome at baseline does not meet the optimistic boundary and does not 
meet WWC representativeness standards. 


8 The evaluators also conducted an interrupted time series with comparison analyses for ELA achievement and 
found not significant impacts. See appendix C. 


° The ELA outcome at follow does not meet the optimistic boundary and does not meet WWC representativeness 
standards. 
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Figure 4: Fidelity of Implementation Rubric Results 
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K} CONCLUSIONS 


3.1 IMPACT STUDY 


In project year 1, we focused on creating a comparison group that met the WWC Evidence 
Standards criteria for the full sample of 4 treatment and 16 comparison schools across the state. 
In project years 2-4 we conducted a series of descriptive statistics to map and graph the school 
outcome data. In project year 5, the final year of grant, we conducted statistical analysis of the 
impact data in addition to mapping and graphing the school outcome data. We found that there 
were no Statistically significant differences in mathematics and ELA outcomes of ELs among 
Exc-EL schools and comparison schools. 


3.2 IMPLEMENTATION STUDY 


Project Exc-EL was implemented with fidelity in year 1. All four project schools had identified 
teacher teams of 4-6 staff members serving ELs and general education students who meet 
regularly as a PLC to examine student progress and implement tiered interventions. Teams also 
focus on embedded professional development and action research using student data to guide 
classroom pedagogy. Project staff, partners, and participants indicated that enhanced 
communications would lead to enhanced collaboration. School staff reported they would like to 
work even more collaboratively with their peers across school and district lines. Community 
partners reported they would like to seek mutually beneficial solutions to reach their shared 
objectives to support EL students and families. 


In year 2, Exc-EL was also implemented with fidelity, but there were concerns regarding 
personalized learning structures and environments (e.g., mentor/mentee programs, student 
advisories, personalized learning plans, student-led parent-teacher conferences, etc.) and the 
timelines for their implementation. The core work of year two focused on the implementation 
and support of New York State Department of Education Commissioner’s Regulations Part 154, 
which requires co-teaching involving EL teachers and core content area teachers. All four project 
schools had core Project Exc-EL PLCs in place. Each team had between 6 and 12 members that 
included core content area teachers, EL teachers, guidance counselors, school administrators, 
school social workers, and additional student support staff. These teams were meeting regularly 
to examine student progress and implement tiered interventions. Teams also focused on 
embedded professional development and action research using student data to guide classroom 
pedagogy. Project staff, partners, and participants indicated that enhanced communications 
between community partners and teachers (regarding the outcomes of 1:1 student support efforts) 
would be beneficial to providing continuous and consistent wraparound supports to students. 


Based on implementation data collections in year 2, it was unclear when some personalized 
learning structures or environments (e.g., student advisory programs, personalized learning 
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plans, etc.) were implemented at all of the four project schools to serve all EL students. Planning 
for these supports and several pilot programs were ongoing. Significant school and district 
staffing changes occurred in year 2. As these were taking place, Project Exc-EL developers 
maintained contact with new school and district leaders to ensure support and collaboration. 
Moving into year 3, project attentions were focused on the continued implementation of project 
components while also considering the impacts of components in place and how best to further 
refine practices. 


The project developer implemented Exc-EL with fidelity in the fourth year of implementation. 
However, there were concerns regarding both a lack of momentum and fragmentation of the 
project at some of the schools involved as the final project year began. The core work of project 
year 4 focused on implementing Dynamic Language Learning Progressions (DLLP) as a follow- 
up to the Summer Institute training provided in August, 2016. However, across the 4 treatment 
schools, the level of DLLP implementation seemed to have lost momentum. At the school level, 
the larger project implementation also lost some energy and focus as the year ended. All 4 
project schools have core Project Exc-EL PLC teams in place, but the number of team members 
and the diversity of the members (in terms of a broad range of school staff—general education, 
EL, administrators, counselors, etc.) seemed to have dropped at some of the schools. While some 
schools reported that diverse teams were meeting regularly to discern and support EL student 
needs using Exc-EL core concepts, others reported that only a few staff members were meeting a 
few times per school year. Project staff, partners, and participants indicated that enhanced 
communications between community partners and teachers proved beneficial to providing 
continuous and consistent wraparound supports to students. 


As was reported in the prior year’s evaluation report, it was unclear from implementation data 
collections when some personalized learning structures or environments (e.g., student advisory 
programs, personalized learning plans, etc.) were implemented at all of the four project schools 
to serve all EL students. Planning for these supports and several pilot programs were still 
ongoing at some schools as the project entered its final year of implementation. School and 
district staffing changes continued to occur. As these took place, Project Exc-EL developers 
worked to consistently maintain contact with new school and district leaders to ensure support 
and collaboration. These efforts were met with varying levels of success, since the level of 
engagement at the district and school leadership levels waned over the fourth project year. 
Moving into project year 5, implementation efforts were fragmenting, as some schools didn’t yet 
feel prepared to implement various project components, while others implemented strategies and 
looked forward to sustaining project components beyond the end of grant funding. 
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The external evaluation of Project Exc-EL included an impact study of student outcomes and an 
implementation study of fidelity of developer objectives. The impact study used quantitative 
methods, relying solely on state assessment data obtained from the New York Department of 
Education. The implementation study used qualitative methods, relying on multiple sources of 
data including annual interviews of key staff, site visits, and document review. In total, the 
external evaluation featured a multi-method approach to understand how the developers 
implemented the intervention and how the intervention affected student outcomes. 


A.1 IMPACT STUDY METHODOLOGY AND DESIGN 


As funded by an Investing in Innovation (13) development grant, Project Exc-EL was a school- 
wide intervention designed to train teachers and counselors, facilitate teacher data teams, and 
provide school instructional coaches. The intent of the project was to improve school supports 
and instruction to ultimately improve EL student outcomes. Because of the school-wide nature of 
the intervention, all teachers, counselors, and principals receive the treatment, and, in turn, all EL 
students received, in theory, improved instruction and supports over the life of the project. 
Hence, the unit of intervention was the school. The long-term outcome, or impact, was English 
learner student achievement on state assessments. The impact study focused on the achievement 
of English learners in the mathematics and English language arts state assessments. 


The evaluation was a quasi-experimental design (QED), where the unit of analysis was the 
school. All data were collected from publicly available annual school report cards, wherein key 
outcomes include mathematics achievement and English language arts (ELA) achievement. 
These achievement scores were state-accountability assessments, where all grades 3-8" students 
and high school students took the state assessments annually. The developer identified the 
treatment schools (N = 4), and the evaluation team selected the comparison schools (N = 16). 
There were no confounds. The treatment and comparison schools were from multiple districts, 
with multiple schools within the treatment and comparison conditions. Characteristics of the 
treatment and comparison schools were similar, except for the use of Project Exc-EL in the 
treatment condition. Time is not a confound since all pre- and post-test data were collected from 
school report cards from the same years for the treatment and comparison schools. 


As indicated in Exhibit A 1, school level data were obtained for the years 2007 through 2017 
through the New York Department of Education. School assessment scores, as well as school 
demographic information, were all publicly available through the school report cards as part of 
the New York State Education Department’s (NYSED’s) annual public reporting. In a review of 
the data, we discerned that while data is available publicly from as early as the 1998-1999 school 
year, the consistent data needed for analysis, such as key school demographic information and 
assessment scores, started from the 2006-2007 school year. Therefore, our pre-intervention data 
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starts from the 2006-2007 school year. Exhibit A 1 indicates the years that were pre-treatment 
and treatment years for the treatment schools. 


Given the multiple years of school-level data, starting with the 2006-2007 school year through 
the 2016-2017 school year (11 years of data), our analytic approach included a descriptive 
approach using a treatment-comparison difference (effect size), and an inferential approach using 
a short interrupted time series with a comparison group (C-ITS) design (Bloom, 2003). We have 
seven years of baseline data and four years of post-intervention data, where the final year of 
treatment data were obtained from the 2016-2017 school year, reflecting the fourth year of the 
five-year 13 grant. 


Exhibit A 1: Treatment Years and Pre-treatment Years for Student Outcomes of Treatment and 
Comparison Schools 


Type of School | 2006- | 2007- | 2008- | 2009- | 2010- | 2011- | 2012- | 2013- | 2014- | 2015- | 2016- | Number 
(Treatment or 2007 | 2008 | 2009 | 2010 | 2011 | 2012 | 2013 | 2014 | 2015 | 2016 | 2017 of 
Comparison) Schools 
Treatment Xx Xx Xx Xx Xx Xx xX T T T T 4 
Comparison Xx Xx Xx Xx X x X c c c c 16 
Total 20 


All achievement scores came from New York State Department of Education assessments administered in the spring of each 
school year. 

“x”: indicates a pre-treatment year when a school outcome score will be obtained. 

“T”: For Treatment schools. 

“c’: For Comparison schools. 


Treatment Schools: Identification, Selection, and Assignment 

The developer identified the treatment schools and recruited the schools during the proposal 
phase in the 2011-2012 school year. In the proposal, there were three districts as part of the 
treatment—Ossining Union Free School District, Tarrytown Union Free School District, and 
White Plains Public Schools. The developer had long-standing partnerships with these districts 
and the superintendent from each district for many years. Upon award, White Plains school 
district withdrew from the project prior to the start of the study with the approval of the US 
Department of Education. Therefore, across two school districts (Ossining and Tarrytown), there 
were four schools in the treatment condition. Both districts, as is the case in many of the districts 
in Westchester County, had one middle school and one high school. Therefore, the four 
treatment schools included the sole middle school and high school in their respective districts. 


Comparison Schools: Identification, Selection, and Assignment 

Across two districts in Westchester County, the developers were working with two middle 
(grades 6-8) schools and two high (grades 9-12) schools. The evaluators identified comparison 
schools for this study by conducting propensity score matching to identify schools and by 
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conducting baseline equivalence testing to ensure the schools were similar in observed 
characteristics to the treatment schools prior to the intervention. Key observed variables included 
percent Limited English Proficient (LEP) students in the school, math achievement for LEP 
students, and English language arts achievement for LEP students. 


The comparison schools serve as “business as usual” conditions. The comparison schools did not 
have Project Exc-EL in their schools during the duration of the study. There was variation across 
the comparison schools in curriculum and instruction, professional development, and college- 
readiness efforts targeted at EL students. However, under the NYSED, curriculum and 
instruction across the state follow the New York State Learning Standards. To graduate from 
high school, all New York students must have a minimum of 22 specific high school credits and 
pass five Regents examinations. 


Our identification process included a series of methods and analyses to ensure baseline 
equivalence, see Appendix B for details. We identified and selected 16 comparison schools, or a 
balance of 1:4 treatment to comparison schools (eight middle school comparison schools, and 
eight high school comparison schools). To identify and select the 16 comparison schools, we first 
identified the matching variables across math outcomes of LEP students and ELA outcomes of 
LEP students. Second, we conducted propensity score matching (PSM) for each outcome 
separately (i.e. two separate PSM for math and ELA), and propensity score matching for all 
outcomes (i.e. one PSM). For the propensity score matching for each outcome, we selected 
different samples of comparison schools. For the propensity score matching for all outcomes, we 
selected one set of comparison schools. Third, we calculated effect sizes for the baseline 
equivalence tests for the achievement outcomes separately, and both outcomes combined. 


Because our confirmatory impact analyses were for all four treatment schools, our primary goal 
was to ensure baseline equivalence, at a minimum, of the confirmatory analytic sample of twenty 
schools (4 treatment and 16 comparison schools).!° We compared baseline equivalence across 
the three sets: 1) ELA outcome of LEP students, 2) Math outcome of LEP students, and 3) All 
outcomes combined. We chose the comparison schools from all outcomes combined because it 
met the WWC Evidence Standards for baseline equivalence, and it was efficient to have a single 
comparison group of schools (rather than separate samples of schools per outcome). 


Impact Study Data Sources 


‘0 We conducted separate analyses to select comparison schools for middle and high schools separately and 
conducted baseline equivalence testing. Our sample was small, with two middle schools matched with eight 
comparison middle schools and two high schools matched with eight comparison high schools. Due to the small 
sample size, we were unable to achieve baseline equivalence that meets WWC standards for schools 
disaggregated by grade level. 
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We collected all administrative (secondary) school-level data from school report cards, as 
published by the NYSED each summer. We downloaded school report cards from the NYSED 
website annually (see: https://data.nysed.gov/downloads.php). NYSED makes this data publicly 
available via Access databases. We downloaded and converted the Access databases into a SAS 
and an R databases for analyses. 


Because of the longitudinal nature of the administrative data, we were able to obtain school-level 
data from the treatment and comparison schools from the 2006-2007 school year, obtaining 
seven years of data prior to Project Exc-EL and four years of data after the initiation of Project 
Exc-EL, through the 2016-2017 school year. 


Although most data for the analysis was obtained from the NYSED website, data from earlier 
years, such as from the 2006-2007 through 2011-2012 school years, were missing data for LEP 
students. The publicly available school report card data suppressed data for LEP students in the 
sample due to the small number of LEP students in each of the schools (note, the intervention 
and treatment was designed for schools with a low-incidence of ELs). Data suppression is based 
on less than 10 students per cell, following US Department of Education guidelines. Therefore, 
the evaluation team worked with NYSED to obtain school-level data of treatment and 
comparison school math and ELA outcomes of LEP students from the 2006-2007 through 2011- 
2012 school years. 


Student Achievement Domain 

Student achievement is measured by the New York state assessment for mathematics and 
English language arts. For middle school students, the achievement measure was the New York 
state math and reading assessments administered each spring. For high school students, the math 
achievement measure was the Regents Integrated Algebra exam administered each spring, and 
the reading achievement measure was the Regents Comprehensive English exam administered 
each spring. These state-wide assessments were not over-aligned with the intervention. The data 
for these math and ELA outcomes were specifically for the subgroup of ELs, taken from the 
school report card. 


The school EL scores were the average scaled scores for all EL students in the school (denoted 
as Limited English Proficient or LEP in the school report card data). These measures were 
consistently collected using the same procedures and rules in both treatment and comparison 
conditions. 


We standardized (z-score) the achievement data. We converted each school’s achievement data 
by grade and by school year, utilizing the standard deviation for the students in that grade, in that 
given school year. We calculated our population standard deviation by including all schools in 
New York that are middle schools (grade 6-8") and high schools (grades 9-12") and using only 
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the schools that had EL test scores by grade level in a given year. For example, a z-score was 
calculated for 6th grade EL students for each school in the 2006-2007 school year using our 
calculated LEP population mean and our calculated LEP population standard deviation as 
denoted in the formula below: 


Where: 

x is the school-level mean from the annual school report card data. For example, this is the 
school-level average of 6" grade EL student mean scores. 

1 is the mean of the population calculated by the authors using NYSED population data. For 
example, this is the population of 6" grade EL student mean scores. 

o is the standard deviation of the population calculated by the authors using NYSED population 
data. For example, this is the population of 6" grade EL student standard deviation. 


To create a middle school score, we first created z-scores for the 6th, 7th, and 8th grades for each 
given year and for each school in the analytic sample. We then created a single score by 
averaging across the z-scores for each grade. As such, each school has grade-specific z-scores, as 
well as an average z-score across grades (1.e., school average) for each year of data. 


For the high school score, students took the Regents Integrated Algebra assessment and the 
Regents Comprehensive English assessment. Students did not take the same algebra or reading 
assessment every year (e.g., 9, 10, 11, and 12™ grades) but rather once during their high 
school experience. Therefore, we created a z-score of the math and reading outcomes to reflect 
the high school scores of all EL students who took the test that school year. 


To create an overall math achievement outcome, we created an average score from the 6th, 7th, 
and 8th grade z-scores and from the Regents Integrated Algebra z-scores. Similarly, to create an 
overall reading achievement outcome, we created an average score from the 6", 7", and 8" grade 
z-scores and from the Regents Comprehensive English z-scores. 


To estimate the impact of Project Exc-EL across education levels, middle schools and high 
schools were analyzed together using the averaged z-scores as the outcome. The math and 
reading outcomes were a common metric for all grades and were interpreted as performance 
relative to the reference population of the LEP students in the state of New York in any given 
year. 
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We also obtained from the school report cards the percent of LEP students in the school to use as 
a covariate. Given that Project Exc-EL focuses on students who were learning English as a new 
language (ENLs), we believe that this is an important covariate to include in the model. 


Impact Study Data Analysis 
We had two confirmatory research questions: 

1) What is the impact of Project Exc-EL on math achievement for middle and high schools 
offered Project Exc-EL for 4 years as compared to middle and high schools in the 
business as usual condition? and 

2) What is the impact of Project Exc-EL on ELA achievement for middle and high schools 
offered Project Exc-EL for 4 years as compared to middle and high schools in the 
business as usual condition? 


The exhibit A 2 below shows the confirmatory contrasts for the achievement domain. The 
confirmatory contrast was used to estimate the impact on school math and reading performance 
for middle and high schools offered Project Exc-EL for four years as compared to middle and 
high schools in the business as usual condition. For the confirmatory analysis, we met baseline 
equivalence with the treatment and comparison schools using school-level data from the 2012- 
2013 school year (one-year prior to the intervention). Baseline equivalence was met for the 
mathematics and English language arts state assessment, and for percent of LEP students in the 
school. See Appendix B for detailed information, and Table B.2 for baseline equivalence 
analysis results. 
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Exhibit A 2: Confirmatory Contrasts for Math and English Language Arts Student Outcomes of Treatment and Comparison Schools 


Treatment Group Comparison Outcome Baseline 
Group 
> s s a a 
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Math QED Project Exc-EL Grades In 2014-15: all grades have [Businessas Achieveme — School: 6th -12'" grade (z- School: Spring 2014 
Achieveme — with 6-12 one year of treatment: Usual (BAU)] nt Standardized scores A. Standardized school A. 6t -12th grade (z-scores 
nt matched All6*— 12th EL school math calculated for math achievement calculated for each grade, 
schools __ students in In 2015-16: 6! graders Students in achievement each grade, pretest combining pooled across grades to 
09/01/2018 treatment schools. have 1 year of treatment, matched combining pooled across NYSTB and Regents generate school level 
Treatment schools 7-42! graders have 2 comparison NYSTB Math grades to averages) 
receive years of treatment. schools and Regents generate school __B. Standardized school 
intervention in level averages) English language arts B. 6" -12'" grade (z-scores 
2014-15, 2015-16, In 2016-17: 6" graders [Continuous] scores from achievement combining calculated for each grade, 
and 2016-17 have 1 year of treatment, 7" three years: NYSTB ELA and pooled across grades to 
school years. graders have 2 years of Spring 2015, Regents generate school level 
treatment, 8-12" graders Spring 2016, and averages) 
have 3 years of treatment. Spring 2017 C. Percent of LEP 
students C. School-level percent in 
2013-2014 school year 
[Continuous] 
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English QED 
Language with 
Arts matched 
Achieveme __ schools 
nt 

09/01/2018 


Project Exc-EL Grades 
6-12 
All 6 — 12th EL 

students in 

treatment schools. 
Treatment schools 

receive 

intervention in 

2014-15, 2015-16, 

and 2016-17 

school years. 


In 2014-15: all grades have 
one year of treatment: 


In 2015-16: 6" graders 
have 1 year of treatment, 
7-42! graders have 2 
years of treatment. 


In 2016-17: 6" graders 
have 1 year of treatment, 7% 
graders have 2 years of 
treatment, 8'"-12t graders 
have 3 years of treatment. 


[BAU] Achieveme 
nt 

Students in 

matched 

comparison 

schools 


School: 
Standardized 
school English 
language arts 
achievement 
combining 
NYSTB ELA and 
Regents 


[Continuous] 


6th -12'" grade (z- 
scores 
calculated for 
each grade, 
pooled across 
grades to 
generate school 
level averages) 
scores from 
three years: 
Spring 2015, 
Spring 2016, and 
Spring 2017 


School: 

A. Standardized school 
English language arts 
achievement combining 
NYSTB ELA and 
Regents 


B. Standardized school 
math achievement 
pretest combining 
NYSTB and Regents 


C. Percent of LEP 
students 


[Continuous] 


Spring 2014 

A. 6 -12t grade (z-scores 
calculated for each grade, 
pooled across grades to 
generate school level 
averages) 


B. 6% -12' grade (z-scores 
calculated for each grade, 
pooled across grades to 
generate school level 
averages) 


C. School-level percent in 
2013-2014 school year 


a These names and expected reporting dates are provided for the AR Team's administrative purposes only and are not recorded for the purposes of assessing scientific process. The names consist of the NEi3 research question 
associated with that contrast and a contrast label. You may choose whether or not to adopt them for your own use. 
b The ‘unit of observation’ is defined as the level at which the data are analyzed. For example, ‘Student’ is listed if each student represents a single case in the dataset (as with individual level state test scores). 'School' is listed if each 
school represents a single case in the dataset (as with school characteristics like AYP or school means of student test scores). 
c The measurement scale describes how the measure is constructed. A measure may be categorized as continuous, ordinal, or binary. Please consult with your TA liaison if you have any questions regarding these measurement 


scales. 
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A.2 IMPLEMENTATION STUDY METHODOLOGY AND DESIGN 


Plus Alpha worked with the project developer to design an implementation study that allows the 
flexibility needed for a development grant while ensuring that fidelity to the key project 
components is defined and assessed across the treatment group (Nelson et al, 2012). Measuring 
Project Exc-EL fidelity began with refining the project logic model provided in the original 
grantee application. The evaluation team worked closely with the intervention developer and the 
evaluation technical assistance provider to develop the logic model (p. 5). The logic model aligns 
with the management plan created by the developer and approved by the USED Program Officer. 
The logic model was used to guide the implementation study design (see Figure 1). Each key 
component consists of indicators of implementation, as can be seen in the Fidelity Matrices for 
each key component (Exhibits 2 — 4). We have provided the operational definition for each 
indicator as well. The same matrix was used for each year during which fidelity was measured. 
Protocols have been created (see Appendix C); each protocol item aligns with an indicator of 
implementation and therefore a key component. Each protocol item is designed to be scored 
either yes or no (0 or 1). Scores roll up to the full sample. We used the Fidelity Matrix to 
measure and assess fidelity for all components and indicators for each of the three years of 
implementation. 
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Exhibit 2: Fidelity Matrix Key Component 1. School climate and structures to support college and career readiness 


Indicator posal Data Sources School Fidelity Full Sample Fidelity 
School coach School coach Evaluator interview with school coach | 0 = Annual needs assessment not 0 = Less than 100% 
conducts needs conducts one using check list protocol conducted of schools meet 
assessment needs 1 = Annual needs assessment school- level 

assessment at School Coaching Form created by the | conducted threshold 
each school evaluator completed by school 1 = 100% of schools 
coaches after each session meet school-level 
threshold 
Review of the Program Officer 
monthly/bi-monthly update provided 
by the developer 
School coach Five (5) coaching | Evaluator interview with school coach | 0 = <3 planned coaching sessions 0 = Less than 100% 
provides sessions are using checklist protocol provided to the school of schools meet 
coaching provided at each 1 = 23 planned coaching sessions school- level 
sessions to the school per year School Coaching Form created by the | provided to the school threshold 
school evaluator completed by school 1 = 100% of schools 
coaches after each session meet school-level 
threshold 
Review of the Program Officer 
monthly/bi-monthly update provided 
by the developer 
Community Developer meets | Evaluator interview with community 0 = Aschool representative does not | 0 =Less than 100% 
partnership quarterly with the | partner organizations using check list | attend each quarterly community of schools meet 
meetings community protocol partnership meeting school- level 
partnership with 1 =A school representative attends threshold 
district and Meeting Form created by the each quarterly community partnership | 1 = 100% of schools 
school evaluator and completed by meeting meet school-level 
representatives partnering organizations and threshold 
present competed after each meeting 
Review of the Program Officer 
monthly/bi-monthly update provided 
by the developer 
Community Developer Evaluator interview with community 0 = Developer does not coordinate 0 = Less than 100% 
partnership coordinates partner organizations using check list | community partnership services at of schools meet 
community protocol each school (less than 100% of the 
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3 Operational eee aie 
Indicator Definition Data Sources School Fidelity Full Sample Fidelity 
service partnership checklist items confirmed during school- level 
coordination services each Meeting Form created by the interview) threshold 


semester at each 
school 


evaluator and completed by 
partnering organizations and 


1 = Developer coordinates community 
partnership services at each school 


1 = 100% of schools 
meet school-level 


competed after each meeting (100% of the checklist items threshold 
confirmed during interview) 
Review of the Program Officer 
monthly/bi-monthly update provided 
by the developer 
Key Component Fidelity Range | 0-4 
Key Component Fidelity Threshold | 4 


Exhibit 3: Fidelity Matrix Key Component 2. Teacher and staff training and technical assistance 


Indicator sora ieg Data Sources School Fidelity Full Sample Fidelity 
Developers Twenty (20) Evaluator interview with developer 0 = <15 hours of instructional practice | 0 = Less than 100% 
provide training hours of and professional development training are provided to each school of schools meet 
on best instructional provider using checklist protocol per year school- level 
instructional practice training 1 = 216 hours of instructional practice | threshold 


practice for ELs 
to school-based 
teams 


are provided to 
each school- 
based team per 
year 


Professional development attendance 
lists collected from the developer 


Professional development debrief 
form created by the evaluator 
completed by the developer after 
each PD session 


Review of the Program Officer 
monthly/bi-monthly update provided 
by the developer 


training are provided to each school 
per year 


1 = 100% of schools 
meet school-level 
threshold 
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Exhibit 4: Fidelity Matrix Key Component 3. Data-driven systematic coaching 


Indicator 


Operational 
Definition 


Data Sources 


School Fidelity 


Full Sample Fidelity 


School based 
teams receive 


Five (5) teacher 
training sessions 


Evaluator interview with developer 
using check list protocol 


0 = <3 planned trainings conducted at 
each school 


0 = Less than 100% 
of schools meet 


training on on Professional 1 = 23 planned trainings conducted at | school- level 
establishing Learning Team meeting attendance lists each school threshold 
Professional Communities are | collected from the developer 1 = 100% of schools 
Learning provided at each meet school-level 
Communities school Team meeting debrief form created threshold 
focused on by the evaluator completed by the 
student data. developer after each PD session. 

Review of the Program Officer 

monthly/bi-monthly update provided 

by the developer 

Key Component Fidelity Range | 0-1 
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Implementation questions (IQ) guided the assessment of fidelity as follows: 
e JQ1 Have the key components of Project Exc-EL been implemented with fidelity? 
e IQ 2 How has implementation varied across the treatment schools in terms of the key project 
components: 
o School climate and structures to support college and career readiness, 
o Teacher and staff training and technical assistance, and 
o Data-driven systemic coaching. 


To guide data gathering in response to the implementation questions, a series of aligning 
documents were created to map from the project logic model to the project management plan 
objectives, strategies, and actions. A fidelity matrix and fidelity indicators were designed and 
aligned with the management plan as well. For the purposes of this development grant 
implementation study, only the project activities within the control of the developer were 
measured, in order to better define and ascertain what implementing Project Exc-EL with high 
fidelity entails. 


Implementation Protocol and Alignment 

The implementation study began with the development of protocols aligned with the project 
management plan, logic model, and evaluation plan. A community partnership form was 
designed to be completed by a community partner member following each community 
partnership meeting. The community partner interview protocol was developed in Year 1, and 
interviews were conducted annually since Year 2 with the same participants (as feasible) to 
ensure consistency. The developer interview protocol was developed in Year | and interviews 
have been conducted annually since Year 2. A school administrator / team leader interview 
protocol was developed in Year 1, and interviews have been conducted annually since Year 2 
with the same participants (as feasible) to ensure consistency. A school coach interview protocol 
was developed in Year 1, and interviews have been conducted annually since Year 2. The school 
coaching activity form was designed in Year 1 to record school coach activities and impressions 
of on-going school coaching throughout the project school year, as completed by school coaches. 
The miscellaneous event protocol was designed to be used by evaluation team members 
attending non-recurring, unplanned, or unscheduled project activities. The final protocol, the 
quarterly management team activity form, was designed to collect information on the quarterly 
project Exc-EL management team meetings, but this form ended up collecting information 
nearly identical to the monthly/bi-monthly Program Officer updates provided by the developer. 
Thus, these forms (completed in full in Year 1 and partially in Year 2) were synthesized and 
analyzed together with the monthly/bi-monthly reports. In Years 3-5, the Program Officer 
updates were used exclusively, in place of the quarterly management team activity form. 


Project protocols were aligned with the Project Management Plan submitted annually to the 
Department of Education. The Project Exc-EL management plan focused on 4 core objectives 
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also found in the logic model and the implementation study fidelity matrix. Each objective was 
further composed of strategies, and each strategy was composed of activities. For example, 
“Objective 1. Improve the capacity of educators to effectively educate ELs within a framework 
of tiered interventions” was supported by four distinct strategies as outlined in the plan, (e.g., 
“Strategy #1.1: Participants on school-based teams participate in training and coaching focused 
on best instructional practices for ELs and effectively incorporate these practices into classroom 
instruction (instructional practices training)”. Strategy 1 was then comprised of eight activities 
(e.g., “Activity 1.1.1 Identify participating schools and educator teams). Following this 
overarching objective, strategy, and activity structure, each study protocol question or item 
mapped back the management plan. The following exhibits detail the alignment of the protocols 
and the management plan objectives (Exhibits 5-7). 
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Exhibit 5: Management Plan Objective 1 Instrument / Protocol Alignment 


Instrument / 
Protocol 


Objective 1. Improve the capacity of educators to effectively 


Strategy #1.1: Participants on 
school-based teams 
participate in training and 
coaching focused on best 
instructional practices for ELs 
and effectively incorporate 
these practices into classroom 
instruction (instructional 
practices training) 


educate ELs within a framework of tiered interventions. 


Strategy #1.2: Participants 
on school-based teams 
participate in training and 
coaching focused on using 
data to personalize 
instruction and intervention 
(tiered intervention training) 


Strategy #1.3: School based 
teams meet 4 times per year 
for coaching and data team 
discussion in order to ensure 
student progress is regularly 
monitored and data is used 
to provide students with 
appropriate supports and 
interventions (data team 
meetings) 


Strategy #1.4: School based 
teams participate in a year- 
end data fair designed to 
promote the sharing of best 
practices and lessons 
learned (dissemination) 


Community Partnership 
Form 


Community Partner 
Interview 


Developer Interview 


v v v v 
School Admin. / Team a Y P V 
Leader Interview 
School Coach Interview V V V 7 
School Coaching Activity V V V V 
Form 
Miscellaneous Event Z V V V 
Protocol 
Quarterly Management 
Team Activity Form 
Program Officer 
monthly/bi-monthly update 
provided by the developer 
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Exhibit 6: Management Plan Objective 2 Instrument / Protocol Alignment 


Objective 2. Structural elements of each school will ensure EL students are part of a smaller 
learning community with a common team of teachers and personalization supports. 


Instrument / 
Protocol 


Strategy #2.1: School based 
teams are formed that include 
core content area teachers, 
ESL, guidance, social worker 
and administrative support. 
Each team works with a 
common set of EL students 
assigned to their team. Teams 
are inclusive of mainstream 
and special needs students, 
and are the same teams 
identified for professional 
development under Obj. #1. 


Strategy #2.2: School-based 
teams meet together and 
focus on student progress 
during regularly scheduled 
common planning time. 


Strategy #2.3: A regular time 
and process for individualized 
student advising (career, 
academic and personal) is 
structured and implemented. 


Strategy #2.4: A process for 
Personal Learning Plan (PLP) 
development and regular use 
by EL students is developed 
and implemented. A critical 
feature of this PLP will be the 
incorporation of student-led 
conferencing. The use of 
digital portfolios will be 
explored as an adjunct use of 
technology. 


Community Partnership 
Form 


Community Partner 
Interview 


Developer Interview 


School Admin. / Team 
Leader Interview 


School Coach Interview 


School Coaching 
Activity Form 


Miscellaneous Event 
Protocol 


Quarterly Management 
Team Activity Form 


Program Officer 
monthly/bi-monthly 
update provided by the 
developer 
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Exhibit 7: Management Plan Objectives 3 and 4 Instrument / Protocol Alignment 


Objective 3. An interagency, inter-district team will be Objective 4. An objective evaluation process 
formed to leverage and share resources and provide will be integrated into project activities to 
support for at-risk EL students and their families. document and improve process and outcome. 
pee , Strategy #4.1: An outside, objective evaluator is 

Strategy #3.1: An inter-agency, Strategy #3.2: Interagency f eas se 
inter-district team will be formed partners will host community alee Sree ba ae Salita Uae ae one 

Instrument / (Project Exc-EL Team) to leverage | meetings to engage families ail Ione ec Ran a cee ee 2 

Protocol resources and provide wrap (topics may include: immigration Garemana are fearniuailibe-ebat a awa tu rier 
around supports for at-risk EL law, assistance with FAFSA, di elecey f : epeige b d 
students and their families. college applications, etc.). FSS UENO) Sea TS MISC ST SSIS le 

: outside organizations. 

Community Partnership a V Y 

Form 

Community Partner Y V y, 

Interview 

Developer Interview P ee Y 

School Admin. / Team e . a 

Leader Interview 

School Coach Interview a 

School Coaching Y V i 

Activity Form 

Miscellaneous Event Y Y P 

Protocol 

Quarterly Management 

Team Activity Form 

Program Officer v v 

monthly/bi-monthly 

update provided by the 

developer 
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Exhibit 8 below shows the key components of the fidelity study cross-walked (1.e., aligned) with 
the fidelity indicators and definitions and the PARC-developed evaluation protocols. 
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Exhibit 8: Implementation Fidelity Matrix Key Components Instrument / Protocol Alignment 


Component 1: School climate and structures to 
support college and career readiness 


Component 2. Teacher 
and staff training and 
technical assistance 


Component 3. Data-driven 
systematic coaching 


School coach : School based teams receive 
Ta School coach provides Community Community PIC loeee provide training on establishing 
AE COTEMES coachin artnershi artnership service siete) On Bes Professional Learnin 
Indicators needs ; " P ti P P di P ti instructional practice for C ities f i 
aacecanent sessions to meetings coordination Elletecchodl baced teams ommunities focused on 
the school student data 
Developer 
; meets quarterly 
School coach a with the nati Twenty (20) hours of eeepc 
Fideli conducts one ane community eogalee instructional practice ves ) teacher gees 
idelity needs sessions are annershin with community Wainina até pravidedtt sessions on Professional 
Definitions provided at ee partnership services Oe Learning Communities are 
assessment at eehnel district and Bach cemesierat each school-based team povided atoach school 
each school aoe school per year p 
per year representatives each school 
present 
Instruments / Protocols 
Community 
Partnership Form if id 
Community ¥ V 
Partner Interview 
Developer V V V V V Vv 
Interview 
School Admin. / 
Team Leader v v v v 
Interview 
School Coach r Y 
Interview 
School Coaching V V V V 
Activity Form 
Miscellaneous V 7 V V V V 
Event Protocol 
Quarterly 
Management 
Team Activity v v 
Form / Program 
Officer update 
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Implementation Study Data Analysis 

We administered the community partner interview check list protocol once per school year, and 
the check list sum was tallied. The community partnership activity form was completed during 
each community partnership activity during each of the implementation years and the resulting 
data was analyzed for content. We administered the developer interview check list protocol once 
per school year, and the check list sum was tallied. The school coaching form was completed 
following each school coaching session during each of the implementation years, and the 
resulting data was analyzed for content. We administered the school coach interview check list 
protocol once per school year, and the check list sum was tallied. The school meeting form was 
completed during each school team meeting session during each of the implementation years, 
and resulting data was analyzed for content. 


Content analysis involved coding the open ended responses using extant codes based on Project 
Exc-EL key components with developer feedback. Emergent codes were used during the coding 
process to provide additional formative feedback to the developer. Two coders coded all 
qualitative data, and a third coder helped to reconcile any coding differences to reach 100% 
agreement in the application of codes. 


We calculate fidelity based on data collected using protocols that we developed (see Appendix 
C). Each protocol item aligned with an indicator, and each indicator aligned with a key 
component. For example, for Project Exc-EL Key Component 2 Teacher and staff training and 
technical assistance to be implemented with fidelity at the school level, 16 or more hours of 
instructional practice training had to be provided to the school each year. We conducted 
interviews with the developer, the school coaches, and the school administrators/team leaders 
using checklist protocols aligned with the management plan activities and professional 
development debrief forms created by the evaluator and completed by the developer after each 
professional development session. These protocols and procedures have been used to determine 
whether or not the key component fidelity threshold was reached each year. See the fidelity 
matrices provided above in Exhibits 2-4. The School Fidelity and Full Sample Fidelity (right) 
columns and the Key Component Fidelity Range and Key Component Fidelity Threshold rows 
(bottom) detail the fidelity score calculation at the key component level. 


Plus Alpha Research & Consulting, LLC 31 


PLUS ALPH 


Consulting, LLC 


PN =)=)=1\1)) @ =u D) =A \| = od (Ole) oto 0) |= WENA INC WV) | Pie) 1 ont |\ (CM Ore) | -y\a1 70) oles 2 (ele) ms} 


The developers identified and recruited the treatment districts and schools. To select comparison 
schools, we used state-wide data from the New York Department of Education to identify a 
matched set of comparison schools. Our identification process included a series of methods and 
analyses to ensure baseline equivalence between four treatment schools and 16 comparison 
schools. To select 16 comparison schools, we followed three steps: 1) Identify matching 
variables, 2) Conduct propensity score matching, and 3) Select a pool of comparison schools by 
conducting baseline equivalence testing to meet WWC standards. Because our confirmatory 
impact analyses were for all four treatment schools, our primary goal was to ensure baseline 
equivalence of the confirmatory analytic sample. 


Step 1: Identifying Matching Variables 
Our matching variables included the outcomes in the achievement domain and the outcome in 
the behavior domain, in addition a key school characteristic, percent of LEP students. The 
following is our list of school-level matching variables: 

e Prior achievement in Math for ELs 

e Prior achievement in ELA for ELs 

e % LEP students 

e % school-wide attendance 
It is important to note the small sample size in these analyses where there were four treatment 
schools (2 high schools and 2 middle schools) to find suitable matched comparison schools. 


Step 2: Conducting Propensity Score Matching 
We employed propensity score matching techniques (PSM) to identify a group of potential 
comparison schools. In this step, we conducted PSM on the following as matching variables: 

e Prior achievement in Math for ELs 

e Prior achievement for ELA for ELs 

e ELA and Math combined (All outcomes). 
For each outcome, we created propensity scores for each school in our sample (treatment and 
comparison). We selected 6-7 comparison schools per treatment school via distance matching. 
To obtain our four comparison schools per treatment school, we then used school outcomes such 
as achievement, attendance, and percent LEP to select the final group of comparison schools. 


We used three different samples of schools. We conducted PSM and identified comparison 
schools within Westchester county, within four contiguous counties surrounding Westchester 
county and New York City (Nassau, Putnam, Suffolk, and Rockland), and state-wide (all schools 
in the state of New York). In essence, we had the following set of comparison schools: 

1. Math outcome: Westchester County 

2. Math outcome: Surrounding Counties 
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Math outcome: Whole State 

ELA outcome: Westchester County 

ELA outcome: Surrounding Counties 

ELA outcome: Whole State 

Combined Math and ELA: Westchester County 
Combined Math and ELA: Surrounding Counties 
Combined Math and ELA: Whole State 


sO SO Os Oe 


Again, given the small number of treatment schools, we first assumed that there would be good 
matches for comparison schools within the same county, Westchester county, given similar sizes 
of school districts and local context. We expanded the potential pool of comparison schools by 
including surrounding contiguous counties, also assuming that these counties share a similar 
local context that surrounds New York City. Finally, we also looked at the whole state for 
matched comparison schools. 


Step 3: Selecting a Pool of Comparison Schools 

We tested for baseline equivalence on all nine samples of comparison schools to determine 
which group of comparison schools were the most like the four treatment schools. We used 
baseline equivalence standards outlined in the What Works Clearinghouse (WWC) Standards and 
Procedures Handbook Version 3. 


For the confirmatory analysis, the target size for the analytic sample was 20 schools, wherein 4 
will be treatment schools and 16 will be comparison schools (8 middle schools and 8 high 
schools). The baseline equivalence testing involved creating an effect size measure for each 
matching variable. For continuous variables, such as school achievement in math and ELA, we 
calculated the effect size based on Hedges’ g. For dichotomous variables, such as school 
attendance rate and percent LEP, we used the Cox’s Index Ratio for Hedge’s g. Our threshold for 
acceptable baseline equivalence, regardless of significance, was ES = .25 following the WWC 
standards!!. 


In comparing the effect sizes across the nine samples, we ultimately chose the sample from the 
combined Math and ELA outcomes and whole state (Choice #9 above). Exhibit B.1 shows the 
descriptive results of the confirmatory analysis sample of comparison and treatment schools. 
Exhibit B.2 shows the baseline equivalence results for the confirmatory analysis sample of 
schools (four treatment and 16 comparison schools). Exhibit B.3 maps the 20 schools in the state 
of New York. Most of the schools are clustered around the Westchester county area, but there are 
a few comparison schools in other metro regions, such as Albany, Buffalo, and Rochester. 


'! Available for download here: 
https://ies.ed.gov/ncee/wwc/Docs/referenceresources/wwc_procedures_v3_0_standards_handbook.pdf 
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Exhibit B 1: Baseline Descriptive Results 


Individual School Performance and Demographic Characteristics 
School Year 2012-2013 (Baseline Equivalence) 


School name 


TREATMENT SCHOOLS 

SLEEPY HOLLOW MIDDLE SCHOOL 
ANNE M DORNER MIDDLE SCHOOL 
SLEEPY HOLLOW HIGH SCHOOL 
OSSINING HIGH SCHOOL 

TOTAL TREATMENT SCHOOLS 
COMPARISON SCHOOLS 

DUNKIRK MIDDLE SCHOOL 
LAWRENCE ROAD MIDDLE SCHOOL 
TURTLE HOOK MIDDLE SCHOOL 

IS 192 THE LINDEN 

SAXTON MIDDLE SCHOOL 

EAST MIDDLE SCHOOL 

SOUTH MIDDLE SCHOOL 

FOX LANE MIDDLE SCHOOL 
WEBSTER-SCHROEDER HIGH SCHOOL 
WESTBURY HIGH SCHOOL 
MANHATTAN BRIDGES HIGH SCHOOL 


INTERNATIONAL HIGH SCHOOL AT 
PROSPECT HEIGHTS 

WALTER G O'CONNELL COPIAGUE HIGH 
SCHOOL 

COMSEWOGUE HIGH SCHOOL 


PATCHOGUE-MEDFORD HIGH SCHOOL 


WESTHAMPTON BEACH SENIOR HIGH 
SCHOOL 
TOTAL 1COMPARISON SCHOOLS 
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Total 
student 
enrollment 


581 
912 
874 
1298 


3,665 


472 
162 
723 
568 
742 
996 
910 
990 
1442 
1262 
555 
395 


1509 


1242 
2486 
1025 


16,079 


Total LEP 
enrollment 


120 
401 


2,059 


% LEP in 
school 


13% 


Math 
(z- 
score) 
for LEP 
students 


-1.23 
-1.33 
0.87 
0.55 
-0.29 


-1.52 
-1.29 
-1.29 
-0.95 
-1.45 
-1.03 
-1.19 
-1.22 
1.13 
-0.11 
1.39 
0.25 


0.94 


0.16 
0.63 
1.09 


-0.28 


ELA 
(z-score) 
for LEP 
students 


-0.90 
-1.18 
0.84 
0.76 
-0.12 


-1.23 
-1.19 
-0.80 
-1.00 
-1.11 
-0.84 
-0.86 
-1.20 
1.92 
0.36 
1.64 
0.86 


1.67 


-0.35 
0.97 
1.14 


0.00 


School 


attendance 


Rate 


96 
97 
95 
97 
96.25 


96 
96 
96 
95 
96 
96 
96 
oh 
96 
94 
92 
90 


96 


96 
95 
97 


95.25 
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Exhibit B 2: Baseline Equivalence Results- Confirmatory Sample School Year 2012-2013 


Contrast Name 


ELA 
Achievement 
for English 
Learners 
Math 
Achievement 
for English 
Learners 
Table Note: 


+ Treatment N 


Treatment 


Unadjusted 
Treatment X 


S 
= 
S) 


-0.29 


jjusted 
Treatment SD 4 


= Unad 


Oo 
ee] 


1.16 


oa Comparison N 


Comparison 

Qa 

I< n 
(= [= 
23 38 
non = no == 
2S 38 
BE BE 
=<) co 
DO DO 
0.00 1.19 
-0.28 1.07 


T-C Difference 


a2 

Go 
se 8 
#298 8 s 
2as an o 
Eao s ° = 
ose 2 ui $ 
reFOa jw a 
-0.12 -0.16 0.63 0.88 


-.01 -0.01 0.65 0.99 


* Standard deviation for the treatment and comparison groups were calculated from state-wide data. 
> The treatment-comparison difference was calculated as the simple difference of unadjusted means. 
° The effect size was calculated using the Hedge’s g formula. 
4 The standard error (s.e.) was calculated using the formula, s.e. =Square root((NttNewi)/ (Nrn*Ncui)+(Hedge’s 
G42)/(2*(NortNewi)))- 


° The p-value was calculated using the formula: p = 2*(1-stdnorm(absolute_value(Hedge’s G/Hedge’s G Standard 


Error))). 


Exhibit B 3: Map of the Confirmatory Sample 
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Representativeness of the Sample at Pre-Test (2012-2013 School Year) 
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While this study focuses on utilizing school-level data, students are nested within schools. 
Therefore, this study is considered a cluster design. In the What Works Clearinghouse (WWC) 
Standards and Procedures Handbook Version 4, representativeness of the sample is a new 
criteria to ensure that the tested students represent the school. Exhibit B.4 shows the total number 
of students in the school, total number of LEP students in the school per start of the school year 
(by the end of September for accountability documentation), and percent LEP tested and the 
number of LEP tested for math and English language arts. As such exhibit B.4 shows the 
representativeness of the treatment and comparison samples at baseline. Exhibit B.5 shows the 
calculations of representativeness for mathematics and B.6 shows the calculations of 
representativeness for ELA. At baseline, we met representativeness for the mathematics outcome 
using the WWC optimistic boundary but did not meet representative for ELA outcome using the 
WWC optimistic boundaries. 


All data are publicly available through the New York State Department of Education, using state 
accountability measures. For overall student enrollment and LEP student enrollment, schools 
collect and report on the data on September 30 or October 1“ of each year, or in the fall semester. 
State assessments, including the score and the number of students who took the assessment, 
occur in May, or in the spring semester. Therefore, while data are all from the same school year, 
it is important to note that enrollment is collected by the school in the fall semester, and state 
assessments data are collected by the school eight months later in spring semester. For specific 
student subgroups, including LEP students (with other subgroups including homeless or migrant 
students), mobility within a single school year can be a factor for student representativeness. In 
addition, federal policies are an important context to the requirements for accountability 
reporting. New York Department of Education received the No Child Left Behind waiver, where 
for the 2012-2013 school year, schools were encouraged to have high percentages of subgroups, 
such as LEP students, tested and reported to the state. 
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Exhibit B 4: Representativeness of the Sample 


Individual School Testing Information: Representativeness of the Sample at Baseline 


School name 


TREATMENT SCHOOLS (N=4) 
SLEEPY HOLLOW MIDDLE SCHOOL 
ANNE M DORNER MIDDLE SCHOOL 
SLEEPY HOLLOW HIGH SCHOOL 
OSSINING HIGH SCHOOL 

TOTAL TREATMENT SCHOOLS 
COMPARISON SCHOOLS (N=16) 
DUNKIRK MIDDLE SCHOOL 
LAWRENCE ROAD MIDDLE SCHOOL 
TURTLE HOOK MIDDLE SCHOOL 

IS 192 THE LINDEN 

SAXTON MIDDLE SCHOOL 

EAST MIDDLE SCHOOL 

SOUTH MIDDLE SCHOOL 

FOX LANE MIDDLE SCHOOL 
WEBSTER-SCHROEDER HIGH SCHOOL 
WESTBURY HIGH SCHOOL 
MANHATTAN BRIDGES HIGH SCHOOL 


INTERNATIONAL HIGH SCHOOL AT 
PROSPECT HEIGHTS 

WALTER G O'CONNELL COPIAGUE HIGH 
SCHOOL 

COMSEWOGUE HIGH SCHOOL 


PATCHOGUE-MEDFORD HIGH SCHOOL 


WESTHAMPTON BEACH SENIOR HIGH 
SCHOOL 
TOTAL COMPARISON SCHOOLS 
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Total 
student 
enrollment 


581 
912 
874 
1298 


3,665 


472 
162 
723 
568 
742 
996 
910 
990 
1442 
1262 
595 
395 


1509 


1242 
2486 
1025 


16,079 


School Year 2012-2013 (Representativeness) 


# of LEP 
Tested in 


Total LEP 
enrollment 


401 


2,059 


% LEP 
Tested in 
Math 


100 
100 
40 
44 


65% 


100 
100 
100 
100 
98 

100 
99 
97 
47 
34 
39 
oil 


50 


46 
38 
38 


60% 


#LEP 
Tested in 
Math 


85 
67 
54 
53 


259 


1241 


% LEP 
Tested in 
ELA 


89 
87 
30 
21 


48% 


94 
97 
89 
11 
84 
89 
94 
88 
47 
28 
47 
39 


29 


42 
32 
22 


54% 


ELA 


71 
58 
40 
25 
194 
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Exhibit B 5: Representativeness Calculation between the Treatment and Control Group for the 
Mathematics Outcome 


Treatment group Control group Overall 
# of EL students enrolled in school in 401 2059 2460 
September 2012 
# of EL students tested in May 2013 259 1241 1500 
Attrition 35.4% 39.7% 39.0% 
Differential attrition 4.3% 


Exhibit B 6: Representativeness Calculation between the Treatment and Control Group for the English 
language arts (ELA) Outcome 


Treatment group Control group Overall 
# of EL students enrolled in school in 401 2059 2460 
September 2012 
# of EL students tested in May 2013 194 1102 1296 
Attrition 51.6% 46.5% 47.3% 
Differential attrition 5.1% 
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APPENDIX C: DETAILED INFORMATION OF THE IMPACT ANALYSIS 


To answer the research questions regarding the impact of Project Exc-EL on student math and 
ELA outcomes, we analyzed the school-level data by first graphically representing the math and 
ELA outcomes by treatment and comparison groups. This graphical representation was helpful to 
understand the data and change over time, but also instrumental in relaying impact information 
to practitioners. Second, given the small sample size driven by the number of treatment schools 
(N = 4 treatment schools), we calculated the treatment and comparison differences and created 
an effect size for math and ELA outcomes using the Hedges’ g formula. Third, given that we had 
11 years of school-level data, we also conducted an interrupted time series with comparison 
analysis. In all three methods, the results were consistent where we did not find significant 
differences in LEP student math or ELA outcomes. 


In addition, the What Works Clearinghouse Standards 4.0 also requires calculations for 
representativeness for cluster design studies. From school report card data collected through the 
New York Department of Education, we documented the number of LEP students, and percent 
and number of LEP students tested in math and ELA at baseline (2012/13 school year) and at 
follow-up (2016/17 school year). Using the optimistic boundary set forth by WWC, while we 
met the representative benchmark for the baseline math outcome, we did not meet 
representativeness for baseline ELA outcome, nor the math and ELA outcomes at follow-up. 


In sum, the impact study is a cluster design that did not meet representativeness standards for 
LEP students, and there were no significant impacts for math and ELA outcomes of LEP 
students. 


Treatment and Comparison Difference Approach (Effect Size) 

The effect size approach to the impact analysis focused on the difference between the baseline 
year (one year prior to the intervention in the 2012/13 school year) and the post-test year (in the 
2016/17 school year). Given the small sample size of schools (4 treatment schools with 16 
matched comparison schools), we first calculated the treatment and comparison group difference 
using the Hedges’ g formula to calculate the effect size. 


The standard deviation is the pooled sample standard deviation for each outcome (Bloom, 2003; 
May et al, 2009). Given that the outcome was standardized using our calculated population 
value, the standard deviation used to calculate the effect size was the pooled standard deviation 
of the standardized (z-score) outcome for the study sample. 


M, — M, 
= ene. 

SDoooled 
Where: 


M, — M,j is the difference between the treatment group mean and comparison group mean; and 
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SDpoolea 18 the weighted pooled standard deviation of the standardized (z-score) outcome for the 


: _ | (4 -1)SD2+(m2—-1)SD3 
study sample, SDjoo1eq = [serpin nens 


Exhibit C.1 shows the effect size for the two confirmatory contrasts. For English language arts 
(ELA) for English learners, the effect size was -0.18 and not significant. For mathematics for 
English learners, the effect size was 0.08 and not significant. 


Exhibit C 1: Impact Estimate of Math and ELA Outcomes for English Learners School Year 2016- 
2017 


Contrast Name Treatment Comparison T -C Difference 
oO ae oO 
Qa = p< wa 7 
| S «4 
= B82 Be) &§ BE BSl#8s 28 : 
oO no oO wo oO = on = n= or ¢. Wp} 
E aye 25 Se eases teal 2 g 
8 88 8S; &€ 8F #8 |/see uj s 
= >= =) (= ro) 530 SO }/FOO ww a. 
ELA 4 -0.50 0.94 16 -0.24 1.13 -0.26 -0.18 0.58 0.77 
Achievement 
for English 
Learners 
Math 4 -0.04 1.61 16 -0.03 17 -0.01 0.08 0.11 0.45 
Achievement 
for English 
Learners 
Table Note: 


* Standard deviation for the treatment and comparison groups were calculated from state-wide data. 
> The treatment-comparison difference was calculated as the simple difference of unadjusted means. 
° The effect size was calculated using the Hedge’s g formula. 


4 The standard error (s.e.) was calculated using the formula, s.e. =Square root((Ntw+tNe)/ (Ntn*Ncui)+(Hedge’s 
G12)/(2*(NtntNeui)))- 


© The p-value was calculated using the formula: p = 2*(1-stdnorm(absolute_value(Hedge’s G/Hedge’s G Standard 
Error))). 


Interrupted Time Series with Comparison Approach 

In addition to the effect size approach, we utilized an interrupted time series with comparison 
approach to take advantage of the 11 years of administrative data from 2006/07 to 2016/17 
school years, where we had seven (7) years of baseline data, and four (4) years of post-test data. 
The comparison interrupted time series (C-SITS) model was analyzed using a linear regression 
model. The C-SITS model adjusts for baseline differences in a way similar to a baseline 
covariate. The model included a dummy variable and interactions of this dummy with the change 


in average test score over time (from before treatment to after treatment was implemented). The 
following is the formal model: 
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Yj; = Bo + Bi(time),; + Bo(group),; + B3(intervention),; + B,y(time x group); 
+ Bs(time since intervention),; + B¢(group x intervention),; 
+ B7(group x time since intervention),; + LEP + &; 


Where Y;; is the outcome for academic achievement scores for EL learners at t= 1, ...., 12 time 
points for math or ELA. Year is a continuous variable indicating the year of measurement; group 
is a dichotomous variable representing treatment (group = 1) versus control schools (group = 0); 
intervention is another dichotomous variable indicating pre- and post-treatment period (1 = post- 
treatment), where the 2016/17 school year was coded as the post-treatment period. The error 
term €,; is assumed to be independent and identically distributed according to a normal 
distribution with a mean of zero and variance 0”. 


Consistent with the simple effect size approach, exhibit C.2 shows the regression results for math 
and English language arts where there were no significant impacts of the intervention. 


Exhibit C 2: Regression Results of the Interrupted Time Series with Comparison (C-SITS) for Math 
and ELA Outcomes for English Learners 


“Parameter SCSC~<“‘<2S; SST Math Achievement SSSCSCSC*C*C*~*C”CSCSC@@CA Achievement 

for English for English 

Learners Learners 

Fixed Est. (S.E.) Est. (S.E.) 
Intercept -0.31 (0.69) -0.50 (2.98) 
Time 0.10 (0.03)* -0.09 (0.19) 
Intervention -0.74 (0.25)* -0.61 (1.17) 
Time Since Intervention 0.08 (0.08) 0.00 (0.41) 
English Learner Percentage 0.02 (0.04) 0.06 (0.15) 
Group 0.00 (0.34) 0.71 (1.57) 
Group*Time -0.07 (0.05) 0.16 (0.29) 
Group*Intervention 0.17 (0.35) -0.21 (1.65) 
Group*Time Since Intervention 0.08 (0.11) -0.18 (0.58) 


Note: The variable for time is centered. * p<.05. 
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Representativeness of the Sample at Post-Test (2016-2017 School Year) 

Exhibit C.3 shows the total number of students in the school, total number of LEP students in the 
school per start of the school year (by the end of September for accountability documentation), 
and percent LEP tested and the number of LEP tested for math and English language arts. As 
such exhibit C.4 shows the representativeness of the treatment and comparison samples at 
follow-up. Exhibit C.4 shows the calculations of representativeness for mathematics and C.5 
shows the calculations of representativeness for ELA. At follow-up, we did not meet 
representativeness for the mathematics outcome using the WWC optimistic boundary and also 
did not meet representative for ELA outcome using the WWC optimistic boundaries. 


All data are publicly available through the New York State Department of Education, using state 
accountability measures. For overall student enrollment and LEP student enrollment, schools 
continue to collect and report on the data on September 30 or October 1“ of each year, or in the 
fall semester. State assessments, including the score and the number of students who took the 
assessment, occur in May, or in the spring semester. Therefore, while data are all from the same 
school year, it is important to note that enrollment is collected by the school in the fall semester, 
and state assessments data are collected by the school eight months later in spring semester. For 
specific student subgroups, including LEP students (with other subgroups including homeless or 
migrant students), mobility within a single school year can be a factor for student 
representativeness. This issue of mobility, especially amongst LEP students, has been noted as an 
issue in recent years from the school staff interviews from the implementation study. In addition, 
federal policies are an important context to the requirements for accountability reporting. It is 
important to note that during the 2016-2017 school year, the Every Student Succeeds Act 
(ESSA) replaced NCLB and the NCLB waiver. Under ESSA, state education agencies and local 
education agencies no longer had specifications to report individual subgroups such as ELs, but 
rather to report for a “high needs” group that included students from low-income families, 
English learners, and students in special education. 
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Exhibit C 3: Representativeness of the Sample 


Individual School Testing Information: Representativeness of the Sample 


School name 


TREATMENT SCHOOLS (N=4) 
SLEEPY HOLLOW MIDDLE SCHOOL 
ANNE M DORNER MIDDLE SCHOOL 
SLEEPY HOLLOW HIGH SCHOOL 
OSSINING HIGH SCHOOL 

TOTAL TREATMENT SCHOOLS 
COMPARISON SCHOOLS (N=16) 
DUNKIRK MIDDLE SCHOOL 
LAWRENCE ROAD MIDDLE SCHOOL 
TURTLE HOOK MIDDLE SCHOOL 

IS 192 THE LINDEN 

SAXTON MIDDLE SCHOOL 

EAST MIDDLE SCHOOL 

SOUTH MIDDLE SCHOOL 

FOX LANE MIDDLE SCHOOL 
WEBSTER-SCHROEDER HIGH SCHOOL 
WESTBURY HIGH SCHOOL 
MANHATTAN BRIDGES HIGH SCHOOL 


INTERNATIONAL HIGH SCHOOL AT 
PROSPECT HEIGHTS 

WALTER G O'CONNELL COPIAGUE HIGH 
SCHOOL 

COMSEWOGUE HIGH SCHOOL 


PATCHOGUE-MEDFORD HIGH SCHOOL 


WESTHAMPTON BEACH SENIOR HIGH 
SCHOOL 
TOTAL COMPARISON SCHOOLS 
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Total 
student 
enrollment 


581 
912 
874 
1298 


3665 


472 
762 
723 
568 
742 
996 
910 
990 
1442 
1262 
595 
395 


1509 


1242 
2486 
1025 


16079 


School Year 2016-2017 (Representativeness) 


# of LEP 
Tested in 


Total LEP 
enrollment 


2692 


% LEP 
Tested in 
Math 


82 
84 
35 
30 


53% 


87 
22 
18 
100 
22 
25 
15 
93 
29 
23 
61 
58 


33 


23 
28 
22 


37% 


#LEP 
Tested in 
Math 


64 
68 
32 
41 


205 


% LEP 
Tested in 
ELA 


68 
15 
25 
35 


ELA 


53 
61 
23 
47 


43 
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Exhibit C 4: Representativeness Calculation between the Treatment and Control Group for the 
Mathematics Outcome 


Treatment group Control group Overall 
# of EL students enrolled in school in 386 2692 3078 
September 2016 
# of EL students tested in May 2017 205 985 1190 
Attrition 46.9% 63.4% 61.3% 
Differential attrition 16.5% 


Exhibit C 5: Representativeness Calculation between the Treatment and Control Group for the English 
language arts (ELA) Outcome 


Treatment group Control group Overall 
# of EL students enrolled in school in 386 2692 3078 
September 2016 
# of EL students tested in May 2017 184 804 988 
Attrition 52.3% 70.1% 67.9% 
Differential attrition 17.8% 
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APPENDIX D: IMPLEMENTATION STUDY PROTOCOLS 


D.1 COMMUNITY PARTNERSHIP ACTIVITY FORM 


Project Exc-EL 1) Location of Activity: 2) Activity Host: 
Community Partnership ae a Se 
Activity Form os 
3) Length of Activity: 4) Activity Date: 5) Activity Time: 
6) When did this group last meet, or | 7) When will this group next meet, 8) This activity occurred: 
when did this activity last occur? or when will this activity occur In person [| By Phone 
again? Via Webinar || As part of another 
event 


9) Activity Participants (Please list name, role, and affiliation): 


10) Activity Topic(s) (Please check all that apply and describe briefly below): 


_| Academic Tutoring _| Adult English language instruction 
Assisting with college applications Assisting with FAFSA completion 
Assisting with immigration law Career awareness 
College awareness Field trip 
Job shadowing Life skills training 
Meeting Mentoring 


11) Briefly outline the community partnership activity. Please list activities, topics, and approximate time spent on 
each. Feel free to share an agenda, notes, minutes, or supporting materials: 


12) Activity Goal(s): 


13) Activity Outcome(s): 


14) Question(s) / Concern(s): 


15) Rate the effectiveness of the Activity (group consensus): 
1 = Little or no learning/effectiveness 
2 = Partial learning or effectiveness 
3 = Adequate group learning or effectiveness 
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D.2 COMMUNITY PARTNER INTERVIEW CHECK LIST PROTOCOL 


Introduction and Purpose 
Plus Alpha Research & Consulting, LLC (Plus Alpha) is completing this interview as part of the implementation 
study of UCLA’s Center X i3 development grant in support of Project Exc-EL. Your responses will help us 


understand: whether or not the key components of Project Exc-EL have been implemented with fidelity and how the 


implementation has varied across the treatment schools in terms of the key project components, such as school 


climate and structures to support college and career readiness, teacher and staff training and technical assistance, and 
data-driven systemic coaching. Your participation is voluntary. You can stop the interview at any time. You will not 
be individually identified in resulting reports. Project Exc-EL evaluation activities have been reviewed and approved 


by UCLA’s Institutional Review Board (IRB). The interview will last approximately 60 minutes. 


Participants 


Project Exc-EL community partners selected by the development team. 


Method 


Interviews are being conducted either in person as part of other project meetings or over the phone. Plus Alpha staff 


will take notes during the interview and will not audio record the interview. Analysts will synthesize notes from 
each interview and include the findings in project reports. The protocol below will be completed by the interviewer 


during the interview. 


Plus Alpha Staff Member(s) Conducting the Interview: 


Date of the interview: 


This interview was conducted: U In person (list location/event): or L) Over the phone 


Start Time: End Time: 


Interview Participant(s) (affiliation, role): 


Project Fidelity! Measures 


Indicator: Definition: Interviewee Involvement and Support(s): 
Community L1 Developer meets quarterly with | LJ Academic tutoring 

partnership the community partnership with CL) Adult English language instruction 
meetings district and school representatives 


present 


XO Assisting with college applications 
LO] Assisting with FAFSA completion 
O Assisting with immigration law 


2 Taken from the Study Design Summary submitted to the US Department of Education as part of the national 


evaluation of the 13 program. These measures are part of project Component 1. School climate and structures to 


support college and career readiness. 
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Indicator: Definition: Interviewee Involvement and Support(s): 
Community CL] Developer coordinates CL] Career awareness 
partnership service | community partnership services 0 College awareness 
egornanon each semester at each school CX Field trip(s) (list purpose and location 
below) 
LI Job shadowing 


XO Life skills training 

CL] Meeting (list type and purpose below) 
L] Mentoring (for whom, how 

C1 Observing a classroom 


CL] Observing a presentation 


Discussion Notes: 


Project Activities" 


Strategy #3.1: An inter-agency, inter-district team will be formed (Project Exc-EL Team) to leverage 
resources and provide wrap around supports for at-risk EL students and their families. 


Activities: Discussion Notes: 


O Activity 3.1.1 Catalog of available 
resources and supports developed 


LO Activity 3.1.2 Project Exc-EL team is 
formed and meets quarterly to 
purposefully match students with services 


L Activity 3.1.3 EL students identified as 
at-risk are offered identified services (i.e., 
tutoring, summer boot camps, family ESL 
classes) 


O Activity 3.1.4 Participation and 
outcomes for all services are monitored 


L) Activity 3.1.5 Evaluate effectiveness of 
community support programs 


Strategy #3.2: Interagency partners will host community meetings to engage families (topics may 
include: immigration law, assistance with FAFSA, college applications, etc.). 


Activities: Discussion Notes: 


'3 Taken from the annual Project Management Plan submitted to the i3 grant funder, the US Department of 
Education. 
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Project Activities’ 


O Activity 3.2.1 Catalog of available 
topics, dates and sites developed 


Additional Discussion Points and Notes: 
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D.3 DEVELOPER INTERVIEW CHECK LIST PROTOCOL 


Introduction and Purpose 

Plus Alpha Research & Consulting, LLC (Plus Alpha) is completing this interview as part of the implementation 
study of UCLA’s Center X i3 development grant in support of Project Exc-EL. Your responses will help us 
understand: whether or not the key components of Project Exc-EL have been implemented with fidelity and how the 
implementation has varied across the treatment schools in terms of the key project components, such as school 
climate and structures to support college and career readiness, teacher and staff training and technical assistance, and 
data-driven systemic coaching. Your participation is voluntary. You can stop the interview at any time. You will not 
be individually identified in resulting reports. Project Exc-EL evaluation activities have been reviewed and approved 
by UCLA’s Institutional Review Board (IRB). The interview will last approximately 60 minutes. 


Participants 
Project Exc-EL project development staff. 


Method 

Interviews are being conducted either in person as part of other project meetings or over the phone. Plus Alpha staff 
will take notes during the interview and will not audio record the interview. Analysts will synthesize notes from 
each interview and include the findings in project reports. The protocol below will be completed by the interviewer 
during the interview. 


Plus Alpha Staff Member(s) Conducting the Interview: 


Date of the interview: 

This interview was conducted: U1 In person (list location/event): or LO) Over the phone 
Start Time: _——sxEnd Time: 

Interview Participant(s) (affiliation, role): 


Project Fidelity! Measures 


Indicator: Definition: Notes: 


Developers provide | 1 Twenty (20) hours of 
training on best instructional practice training are 


instructional provided to each school-based 
practice for ELs to | team per year 


school-based 
teams). 


4 Taken from the Study Design Summary submitted to the US Department of Education as part of the national 
evaluation of the 13 program. 
'S This measure is part of Component 2. Teacher and staff training and technical assistance. 
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Indicator: Definition: 


Notes: 


School based teams | 1] Five (5) teacher training 
receive training on sessions on Professional Learning 
establishing Communities are provided at 


Professional each school 
Learning 
Communities 
focused on student 
data!®. 


Discussion Notes: 


Project Activities'’ 


Strategy #1.1: Participants on school-based teams participate in training and coaching focused on best 
instructional practices for ELs and effectively incorporate these practices into classroom instruction 


(instructional practices training) 


Activities: 


Discussion Notes: 


QO Activity 1.1.1 Identify participating 
schools and educator teams 


QO Activity 1.1.2 Roll out project at 
participating schools 


LO Activity 1.1.3 Conduct a readiness 
assessment for educators to determine 
state of current knowledge and practice 


LO Activity 1.1.4 Create a plan for training 
that includes content and logistics 


O Activity 1.1.5 Conduct training 


LO Activity 1.1.6 Conduct site-based 
coaching 4 times per year 


LO Activity 1.1.7 Evaluate usefulness and 
impact of summer training 


L] Activity 1.1.8 Evaluate usefulness and 
impact of coaching 


Strategy #1.2: Participants on school-based teams participate in training and coaching focused on using 
data to personalize instruction and intervention (tiered intervention training) 


Activities: 


Discussion Notes: 


‘6 This measure is part of Component 3. Data-driven systematic coaching. 
7 Taken from the annual Project Management Plan submitted to the i3 grant funder, the US Department of 


Education. 
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Project Activities'” 


L Activity 1.2.1] Conduct a readiness 
assessment for educators to determine 
state of current knowledge and practice 
(done in conjunction with Activity 1.1.3) 


U1 Activity 1.2.2 Create a plan for training 
that includes content and logistics 


O Activity 1.2.3 Conduct training 


CL Activity 1.2.4 Conduct site-based data 
team meetings 4 times per year 


L] Activity 1.2.5 Evaluate usefulness and 
impact of summer training 


C1 Activity 1.2.6 Evaluate usefulness and 
impact of data team meetings 


Strategy #1.3: School based teams meet 4 times per year for coaching and data team discussion in order 
to ensure student progress is regularly monitored and data is used to provide students with appropriate 
supports and interventions (data team meetings) 


Activities: 


Discussion Notes: 


CL Activity 1.3.1 Create a schedule that 
allows site based teams to meet 4 times 
per year for at least 90 minutes per 
meeting 


LO Activity 1.3.2 Create a protocol that 
allows site based teams to effectively and 
efficiently use student data to identify 
student progress and create appropriate 
interventions 


1 Activity 1.3.3 Conduct site-based team 
meetings 


Strategy #1.4: School based teams participate in a year-end data fair designed to promote the sharing of 
best practices and lessons learned (dissemination) 


Activities: 


Discussion Notes: 


QO Activity 1.4.1 Create a plan for a year 
end data fair that includes logistics that 
allows all teams to participate 


C1 Activity 1.4.2 Create a protocol that 
allows site based teams to share their 
lessons learned 


LO Activity 1.4.3 Conduct the year end 
data fair 
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Project Activities'” 


CL Activity 1.4.4 Develop an electronic 
platform that will store and facilitate 
sharing of best practices, lessons and 
lessons learned 


CL Activity 1.4.5 Populate electronic 
platform with materials developed by site- 
based teams 


CL] Activity 1.4.6 Create a strategy for 
widely sharing and promoting the use of 
the electronic platform materials 


CL] Activity 1.4.7 Disseminate lessons 
learned 


Strategy #2.1: School based teams are formed that include core content area teachers, ESL, guidance, 


social worker, and administrative support. E 


ach team works with a common set of EL students assigned 


to their team. Teams are inclusive of mainstream and special needs students, and are the same teams 
identified for professional development under Objective #1. 


Activities: 


Discussion Notes: 


L Activity 2.1.1 Plan for a school 
readiness assessment 


L Activity 2.1.2 Conduct school readiness 
assessment 


LO Activity 2.1.3 EL students are 
scheduled and assigned to teams 


LO Activity 2.1.4 Evaluate the ability to 
create effective teams 


Strategy #2.2: School-based teams meet together and focus on student progress during regularly 


scheduled common planning time 


Activities: 


Discussion Notes: 


CL] Activity 2.2.1 Coaches meet with 
administrators, school leadership teams 
and others to review the schedule and 
revise as necessary. 


UL) Activity 2.2.2 Evaluate the 
implementation and impact of common 
planning time 


Strategy #2.3: A regular time and process fo 
personal) is structured and implemented 


r individualized student advising (career, academic and 


Activities: 


Discussion Notes: 


TL) Activity 2.3.1 Coaches meet with 
administrators, school leadership teams 
and others to develop needed structures 
and processes 
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Project Activities'” 


LO Activity 2.3.2 Evaluate the 
development of a student advisory model 


Strategy #2.4: A process for Personal Learning Plan (PLP) development and regular use by EL students 
is developed and implemented. A critical feature of this PLP will be the incorporation of student-led 
conferencing. The use of digital portfolios will be explored as an adjunct use of technology 


Activities: 


Discussion Notes: 


L] Activity 2.4.1 Coaches meet with 
administrators, school leadership teams 
and others to develop needed structures 
and processes 


L) Activity 2.4.2 Evaluate the 
development of a PLP model 


Strategy #3.1: An inter-agency, inter-district team will be formed (Project Exc-EL Team) to leverage 


resources and provide wrap around supports 


for at-risk EL students and their families 


Activities: 


Discussion Notes: 


O Activity 3.1.1 Catalog of available 
resources and supports developed 


LO Activity 3.1.2 Project Exc-EL team is 
formed and meets quarterly to 
purposefully match students with services 


CL Activity 3.1.3 EL students identified as 
at-risk are offered identified services (i.e., 
tutoring, summer boot camps, family ESL 
classes) 


O1 Activity 3.1.4 Participation and 
outcomes for all services are monitored 


LO Activity 3.1.5 Evaluate effectiveness of 
community support programs 


Strategy #3.2: Interagency partners will host community meetings to engage families (topics may 
include: immigration law, assistance with FAFSA, college applications, etc.). 


O Activity 3.2.1 Catalog of available 
topics, dates, and sites developed. 


Additional Discussion Points and Notes: 
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D.4 PROJECT SCHOOL ADMINISTRATOR OR SCHOOL DATA TEAM LEAD CHECK LIST PROTOCOL 


Introduction and Purpose 

Plus Alpha Research & Consulting, LLC (Plus Alpha) is completing this interview as part of the implementation 
study of UCLA’s Center X i3 development grant in support of Project Exc-EL. Your responses will help us 
understand: whether or not the key components of Project Exc-EL have been implemented with fidelity and how the 
implementation has varied across the treatment schools in terms of the key project components, such as school 
climate and structures to support college and career readiness, teacher and staff training and technical assistance, and 
data-driven systemic coaching. Your participation is voluntary. You can stop the interview at any time. You will not 
be individually identified in resulting reports. Project Exc-EL evaluation activities have been reviewed and approved 
by UCLA’s Institutional Review Board (IRB). The interview will last approximately 60 minutes. 


Participants 
Project Exc-EL school administrator or data team lead. 


Method 

Interviews are being conducted either in person as part of other project meetings or over the phone. Plus Alpha staff 
will take notes during the interview and will not audio record the interview. Analysts will synthesize notes from 
each interview and include the findings in project reports. The protocol below will be completed by the interviewer 
during the interview. 


Plus Alpha Staff Member(s) Conducting the Interview: 

Date of the interview: 

This interview was conducted: U In person (list location/event): or LO) Over the phone 
Start Time:___——_=—sxEnd Time:_____ 

Interview Participant(s) (affiliation, role): 


Project Fidelity!’ Measures 


Indicator: Definition: Notes: 


School based teams | 1) Five (5) teacher training 
receive training on | sessions on Professional Learning 
establishing Communities are provided at 
Professional each school 

Learning 
Communities 
focused on student 
data. 


'8 Taken from the Study Design Summary submitted to the US Department of Education as part of the national 
evaluation of the 13 program. This measure is part of Component 3. Data-driven systematic coaching. 
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Indicator: Definition: 


Notes: 


Discussion Notes: 


Project Activities’? 


Strategy #1.1: Participants on school-based teams participate in training and coaching focused on best 
instructional practices for ELs and effectively incorporate these practices into classroom instruction 


(instructional practices training) 


Activities: 


Discussion Notes: 


O Activity 1.1.1 Identify participating 
schools and educator teams 


O Activity 1.1.2 Roll out project at 
participating schools 


LO Activity 1.1.3 Conduct a readiness 
assessment for educators to determine 
state of current knowledge and practice 


1 Activity 1.1.6 Conduct site-based 
coaching 4 times per year 


LO Activity 1.1.7 Evaluate usefulness and 
impact of summer training 


XL Activity 1.1.8 Evaluate usefulness and 
impact of coaching 


Strategy #1.2: Participants on school-based teams participate in training and coaching focused on using 
data to personalize instruction and intervention (tiered intervention training) 


Activities: 


Discussion Notes: 


L Activity 1.2.1 Conduct a readiness 
assessment for educators to determine 
state of current knowledge and practice 
(done in conjunction with Activity 1.1.3) 


R 


O Activity 1.2.4 Conduct site-based data 
team meetings 4 times per year 


LO Activity 1.2.5 Evaluate usefulness and 
impact of summer training 


XL Activity 1.2.6 Evaluate usefulness and 
impact of data team meetings 


Strategy #1.3: School based teams meet 4 times per year for coaching and data team discussion in order 
to ensure student progress is regularly monitored and data is used to provide students with appropriate 
supports and interventions (data team meetings) 


'? Taken from the annual Project Management Plan submitted to the i3 grant funder, the US Department of 


Education. 
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Project Activities’? 


Activities: 


Discussion Notes: 


CL Activity 1.3.1 Create a schedule that 
allows site based teams to meet 4 times 
per year for at least 90 minutes per 
meeting 


QO Activity 1.3.2 Create a protocol that 
allows site based teams to effectively and 
efficiently use student data to identify 
student progress and create appropriate 
interventions 


LO Activity 1.3.3 Conduct site-based team 
meetings 


R 


Strategy #1.4: School based teams participate in a year-end data fair designed to promote the sharing of 
best practices and lessons learned (dissemination) 


Activities: 


Discussion Notes: 


1 Activity 1.4.] Create a plan for a year 
end data fair that includes logistics that 
allows all teams to participate 


O Activity 1.4.2 Create a protocol that 
allows site based teams to share their 
lessons learned 


LO Activity 1.4.3 Conduct the year end 
data fair 


CL Activity 1.4.4 Develop an electronic 
platform that will store and facilitate 
sharing of best practices, lessons and 
lessons learned 


QO Activity 1.4.5 Populate electronic 
platform with materials developed by site- 
based teams 


CL Activity 1.4.6 Create a strategy for 
widely sharing and promoting the use of 
the electronic platform materials 


L] Activity 1.4.7 Disseminate lessons 
learned 


Strategy #2.1: School based teams are formed that include core content area teachers, ESL, guidance, 
social worker, and administrative support. Each team works with a common set of EL students assigned 
to their team. Teams are inclusive of mainstream and special needs students, and are the same teams 
identified for professional development under Objective #1. 


Activities: 


Discussion Notes: 


LO Activity 2.1.1 Plan for a school 
readiness assessment 
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Project Activities’? 


LO Activity 2.1.2 Conduct school readiness 
assessment 


L Activity 2.1.3 EL students are 
scheduled and assigned to teams 


LO Activity 2.1.4 Evaluate the ability to 
create effective teams 


Strategy #2.2: School-based teams meet together and focus on student progress during regularly 


scheduled common planning time 


Activities: 


Discussion Notes: 


L1 Activity 2.2.1 Coaches meet with 
administrators, school leadership teams 
and others to review the schedule and 
revise as necessary. 


L Activity 2.2.2 Evaluate the 
implementation and impact of common 
planning time 


Strategy #2.3: A regular time and process for individualized student advising (career, academic and 


personal) is structured and implemented 


Activities: 


Discussion Notes: 


CL] Activity 2.3.1 Coaches meet with 
administrators, school leadership teams 
and others to develop needed structures 
and processes 


UL) Activity 2.3.2 Evaluate the 
development of a student advisory model 


Strategy #2.4: A process for Personal Learning Plan (PLP) development and regular use by EL students 
is developed and implemented. A critical feature of this PLP will be the incorporation of student-led 
conferencing. The use of digital portfolios will be explored as an adjunct use of technology 


Activities: 


Discussion Notes: 


CL) Activity 2.4.1 Coaches meet with 
administrators, school leadership teams 
and others to develop needed structures 
and processes 


XL Activity 2.4.2 Evaluate the 
development of a PLP model 


Strategy #3.1: An inter-agency, inter-district team will be formed (Project Exc-EL Team) to leverage 


resources and provide wrap around supports 


for at-risk EL students and their families 


Activities: 


Discussion Notes: 


O Activity 3.1.1 Catalog of available 
resources and supports developed 


LO Activity 3.1.2 Project Exc-EL team is 
formed and meets quarterly to 


purposefully match students with services 
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Project Activities!” 


L] Activity 3.1.3 EL students identified as 
at-risk are offered identified services (i.e., 
tutoring, summer boot camps, family ESL 
classes) 


QO Activity 3.1.4 Participation and 
outcomes for all services are monitored 


L Activity 3.1.5 Evaluate effectiveness of 
community support programs 


Strategy #3.2: Interagency partners will host community meetings to engage families (topics may 
include: immigration law, assistance with FAFSA, college applications, etc.). 


O Activity 3.2.1 Catalog of available 
topics, dates, and sites developed. 


Additional Discussion Points and Notes: 
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D.5 SCHOOL COACH INTERVIEW CHECK LIST PROTOCOL 


Introduction and Purpose 

Plus Alpha Research & Consulting, LLC (Plus Alpha) is completing this interview as part of the implementation 
study of UCLA’s Center X i3 development grant in support of Project Exc-EL. Your responses will help us 
understand: whether or not the key components of Project Exc-EL have been implemented with fidelity and how the 
implementation has varied across the treatment schools in terms of the key project components, such as school 
climate and structures to support college and career readiness, teacher and staff training and technical assistance, and 
data-driven systemic coaching. Your participation is voluntary. You can stop the interview at any time. You will not 
be individually identified in resulting reports. Project Exc-EL evaluation activities have been reviewed and approved 
by UCLA’s Institutional Review Board (IRB). The interview will last approximately 60 minutes. 


Participants 
Project Exc-EL school coaching staff. 


Method 

Interviews are being conducted either in person as part of other project meetings or over the phone. Plus Alpha staff 
will take notes during the interview and will not audio record the interview. Analysts will synthesize notes from 
each interview and include the findings in project reports. The protocol below will be completed by the interviewer 
during the interview. 


Plus Alpha Staff Member(s) Conducting the Interview: 

Date of the interview: 

This interview was conducted: U In person (list location/event): or LK Over the phone 
Start Time:__——sxEnd Time: 


Interview Participant(s) (affiliation, role): 


Project Fidelity?" Measures 


Indicator: Definition: Notes: 
School coach L] School coach conducts one 

conducts needs needs assessment at each school 
assessment 

School coach L] Five (5) coaching sessions are 
provides coaching provided at each school per year 

sessions to the 

school 


°Taken from the Study Design Summary submitted to the US Department of Education as part of the national 
evaluation of the 13 program. These measures are part of project Component 1. School climate and structures to 
support college and career readiness. 
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Indicator: Definition: Notes: 


Discussion Notes: 


Project Activities”! 


Strategy #1.1: Participants on school-based teams participate in training and coaching focused on best 
instructional practices for ELs and effectively incorporate these practices into classroom instruction 
(instructional practices training) 


Activities: Discussion Notes: 


L Activity 1.1.1 Identify participating 
schools and educator teams 


QO Activity 1.1.2 Roll out project at 
participating schools 


LO Activity 1.1.3 Conduct a readiness 
assessment for educators to determine 
state of current knowledge and practice 


O Activity 1.1.4 Create a plan for training 
that includes content and logistics 


O Activity 1.1.5 Conduct training 


L Activity 1.1.6 Conduct site-based 
coaching 4 times per year 


LO Activity 1.1.7 Evaluate usefulness and 
impact of summer training 


LO Activity 1.1.8 Evaluate usefulness and 
impact of coaching 


Strategy #1.2: Participants on school-based teams participate in training and coaching focused on using 
data to personalize instruction and intervention (tiered intervention training) 


Activities: Discussion Notes: 


CL Activity 1.2.1 Conduct a readiness 
assessment for educators to determine 
state of current knowledge and practice 
(done in conjunction with Activity 1.1.3) 


QO Activity 1.2.2 Create a plan for training 
that includes content and logistics 


O Activity 1.2.3 Conduct training 


L Activity 1.2.4 Conduct site-based data 
team meetings 4 times per year 


*! Taken from the annual Project Management Plan submitted to the i3 grant funder, the US Department of 
Education. 
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Project Activities”! 


KL Activity 1.2.5 Evaluate usefulness and 
impact of summer training 


XL Activity 1.2.6 Evaluate usefulness and 
impact of data team meetings 


Strategy #1.3: School based teams meet 4 times per year for coaching and data team discussion in order 
to ensure student progress is regularly monitored and data is used to provide students with appropriate 
supports and interventions (data team meetings) 


Activities: 


Discussion Notes: 


XL Activity 1.3.1 Create a schedule that 
allows site based teams to meet 4 times 
per year for at least 90 minutes per 
meeting 


QO Activity 1.3.2 Create a protocol that 
allows site based teams to effectively and 
efficiently use student data to identify 
student progress and create appropriate 
interventions 


LO Activity 1.3.3 Conduct site-based team 
meetings 


Strategy #1.4: School based teams participate in a year-end data fair designed to promote the sharing of 
best practices and lessons learned (dissemination) 


Activities: 


Discussion Notes: 


O1 Activity 1.4.1 Create a plan for a year 
end data fair that includes logistics that 
allows all teams to participate 


LO Activity 1.4.2 Create a protocol that 
allows site based teams to share their 
lessons learned 


LO Activity 1.4.3 Conduct the year end 
data fair 


CL Activity 1.4.4 Develop an electronic 
platform that will store and facilitate 
sharing of best practices, lessons and 
lessons learned 


1 Activity 1.4.5 Populate electronic 
platform with materials developed by site- 
based teams 


CL Activity 1.4.6 Create a strategy for 
widely sharing and promoting the use of 
the electronic platform materials 


CL Activity 1.4.7 Disseminate lessons 
learned 
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Project Activities”! 


Strategy #2.1: School based teams are formed that include core content area teachers, ESL, guidance, 


social worker, and administrative support. E 


ach team works with a common set of EL students assigned 


to their team. Teams are inclusive of mainstream and special needs students, and are the same teams 
identified for professional development under Objective #1. 


Activities: 


Discussion Notes: 


KL Activity 2.1.1 Plan for a school 
readiness assessment 


CL Activity 2.1.2 Conduct school readiness 
assessment 


XL Activity 2.1.3 EL students are 
scheduled and assigned to teams 


QO Activity 2.1.4 Evaluate the ability to 
create effective teams 


Strategy #2.2: School-based teams meet together and focus on student progress during regularly 


scheduled common planning time 


Activities: 


Discussion Notes: 


CL) Activity 2.2.1 Coaches meet with 
administrators, school leadership teams 
and others to review the schedule and 
revise as necessary. 


L Activity 2.2.2 Evaluate the 
implementation and impact of common 
planning time 


Strategy #2.3: A regular time and process fo 
personal) is structured and implemented 


r individualized student advising (career, academic and 


Activities: 


Discussion Notes: 


CL Activity 2.3.1 Coaches meet with 
administrators, school leadership teams 
and others to develop needed structures 
and processes 


L) Activity 2.3.2 Evaluate the 
development of a student advisory model 


Strategy #2.4: A process for Personal Learning Plan (PLP) development and regular use by EL students 
is developed and implemented. A critical feature of this PLP will be the incorporation of student-led 
conferencing. The use of digital portfolios will be explored as an adjunct use of technology 


Activities: 


Discussion Notes: 


CL Activity 2.4.1 Coaches meet with 
administrators, school leadership teams 
and others to develop needed structures 
and processes 


LO Activity 2.4.2 Evaluate the 
development of a PLP model 
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Additional Discussion Points and Notes: 
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D.6 


SCHOOL COACHING ACTIVITY FORM 


Project Exc-EL 
School Coaching Form 


1) School: 


2) Length of Coaching Session: 


3) Coach: 


4) Coach’s Affiliation: 


5) Date: 


6) Participants in Coaching Session (list staff member names and roles): 


7) Coaching Topic(s) 

(check all that apply): 
Advising for Success: 

--|| Individual Student Focus 
--/| Small Group Focus 
Personalized Learning Plans 
(PLPs) 

Response to Intervention (RtI) 
School schedule 

Sheltered Instruction 
Observation Protocol (SIOP) 
Supporting EL students 
Teacher Data Team 

Other (describe): 


&) Coaching Session Frequency 
(with this specific individual or 
group): 

Weekly 

Monthly 

Each Semester 

Annually 

Other (describe): 


9) When did your last coaching 
session with this individual or 
group occur? 


10) This coaching session 
occurred 
(check all that apply): 
In person / face-to-face 
Over the phone / via conference 
call 
Virtually—via a webinars, etc. 
In conjunction with another 
event 

(i.e. a conference, another 
meeting, 

etc.) 
Other (describe): 


11) When is your next coaching 
session scheduled to occur with 
this individual or group? 


12) Briefly outline the coaching session (list activities, topics, and approximate time spent on each): 


13) Coaching Session Goal(s): 


14) Coaching Session Outcome(s): 


15) Question(s) / Concern(s): 


16) Rate the effectiveness of the coaching session (group consensus): 
1 = Little or no learning/effectiveness 
2 = Partial learning or effectiveness 
3 = Adequate group learning or effectiveness 
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D.7 MISCELLANEOUS PROJECT EVENT PROTOCOL 


This protocol is a catch all for activities and events not already covered by other project protocols or data collection 


efforts. As such, this protocol should not be used to record the proceedings of a Community Partnership Activity 


(this data is collected by the developer using the Community Partnership Activity form), School Coaching, (this data 
is collected by the coaches using the School Coaching form), School (Team) Meetings (this data is collected by the 


school team using the Scholl Meeting form), or a Quarterly Project Partnership Activity (this data is collected by 


partners using the Quarterly Activity Summary Report). This protocol should be used to record school observations, 


impromptu conversations, impromptu project events, etc. 


Please attach or include supporting documents or related resources when sending this completed protocol. 


Role of the person completing this form: 
CO Evaluation Team Member CL] Development Team Member 
CO Other role, briefly describe: 


Date of the activity: 

How did you attend this event? U In person, UH By phone, U1 Via Webinar, UH As part of another event 
Location of the Activity, briefly describe: 

Start Time: —sxEnd Time: 

Event Frequency: U Recurring Event or 0 One-Time Event 


Activity Participants (Please list name, role, and affiliation): 


Activity Topic(s) (Please check all that apply and describe briefly below): 

LC) Academic Tutoring CL) Adult English language instruction 
LO Assisting with college applications C1 Assisting with FAFSA completion 
XO Assisting with immigration law CL] Career awareness 

LI College awareness XO Field trip 

XO Job shadowing XO Life skills training 

C1 Meeting LC) Mentoring 

LC Observing a classroom CL Observing a presentation 


Activity Description (a brief paragraph): 


Activity Goals and Outcomes (if applicable): 
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D.8 QUARTERLY MANAGEMENT TEAM ACTIVITY FORM 


[redundant with the Program Officer updates and superseded as such in year 2] 


School or Partner Name: 


Date Activity Summary Sheet Value/Action 
Attached? 
Yes (| No $ 
Paid 
To be invoiced 
In Kind 
Yes || No $ 
Paid 
To be invoiced 
In Kind 
Yes (| No $ 
(| Paid 
To be invoiced 
In Kind 
Yes || No $ 
Paid 
To be invoiced 
In Kind 
Yes || No $ 
Paid 
To be invoiced 
In Kind 
Yes (| No $ 
Paid 
To be invoiced 
In Kind 
Briefly summarize the activities for the period covered and why you believe they were 
successful: 
Briefly summarize any challenges or barriers you encountered, including suggestions for 
mitigation: 
Other comments or suggestions: 
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